CONTACTLESS H UMAN ACTIVITY RECOGNITION By Brandon Lamont Harrington A THESIS S ubmitted to Michigan State University in partial fulfillment of the requirements for the degree of Electrical Engineering - Master of Science 2020 ABSTRACT CONTACTLESS H UMAN ACTIVITY RECOGNITION By Brandon Lamont Harrington The ob jective of this thesis is to de sign passive measurement and classification system s to recognize various human activities. Human Activ ity Recognition (HAR) can enhance a diverse range of human - centri c applications in health care, smart homes, and security. Traditional solutions are based on wearable sensors and vision - based technologies; however, these solutions suffer from considerable limitations . T he need for users to wear sensors for activity r eco gnitio n is inconvenient and impractical for long periods of time. Contactless HAR systems have increased the abilities, practicali ty, and c onvenience of sensor based HAR systems. It allows for unobtrusive measurement and classification of seden tary behavio rs in the workplace such as time spent sitting at desk, to intense physical activi ties such as at home exercises . T hi s thesis provides contributions to contactless HAR methods and techniques using di fferent sensing modalities . While activity recognition ca n be ach ieved with low cost ultrasound sensors, we show the beneficial impact of adopting radio frequency as a technologic al means of recognizing human activity. Specifically, WiFi signal analys i s enable s both macro and micro level activity classification. The ubi quity of W iFi infrastructures in home, university, work, and even outdoor enviro nments makes WiFi the most conveni ent technology to produc e va luable contributions in human activity recognition. The primary contributions of this work are: 1) Develo pment of prototype hardware system , Echolocation Activity Detect or (EAD), that achieves c on tactless activity recognition in office setting. 2) A low - cost system that employs WiFi m o nitoring of packets to provide a student engagement measure , and 3) De s ign of a non - invasive system that recognizes exercise activity and p rovides fine - grained repetition counting i nformation of each exercise set using WiFi channel state information . E ach of the contribut ions passively detect s and classif ies human activity in a c o ntactless fashion without the need of wearable sensors. iii ACKNOWLEDGEMENTS I would like t o thank my advisor, Dr. Subir Biswas , for his mento r ship throughout my graduate carrer. From him I learned how to be curious, optimistic and enthusiastic, and how to t hink independently . I would also like to thank my committee Dr. Nih ar Mahapatra and Dr. Nelson Sepulv eda for their time a n d guidance. My thanks a lso go to past and current members of our lab: Faezeh Hajiaghajani , Saptarshi Das, Ian Shi, Feng Dezhi, Rui Wa ng, Henry Griffith and Shahrukh Kasi Khan for their collaboration, s u pport and for many memor able moments. Additionally , I would like to that Steven Thomas, Katy Colby, and Mea gan Kroll for their tremendous l ove , support and advice. Last but not least, I w ould like to give thanks to my parents and siblings for making sure that I feel their love and support through tough times. iv T ABLE OF CONTENTS LIST OF TABLES ................................ ................................ ................................ ................................ ....... vi LIST OF FIGURES ................................ ................................ ................................ ................................ .... vi i 1. Int roduction ................................ ................................ ................................ ................................ ... 1 1.1. Focus of Research ................................ ................................ ................................ ................. 1 1.2. Contactless Sensing Technologies ................................ ................................ ........................ 2 1.2.1. Ultrasound ................................ ................................ ................................ ......................... 2 1.2.2. Radio Frequency (RF) ................................ ................................ ................................ ....... 2 1.3. Sensing Techniques ................................ ................................ ................................ ............... 4 1.3.1. First - Reflection Echolocati on ................................ ................................ ........................... 4 1.3.2. Received Sig nal Strength Indicator (RSSI) ................................ ................................ ....... 4 1.3.3. Channel State Informat ion (CSI) ................................ ................................ ...................... 5 1.4. Research Objectives ................................ ................................ ................................ .............. 7 1.4.1 Human a ctivity analysis using ultrasound ................................ ................................ ......... 7 1.4.2 Student engagement meas urement using Wi - Fi RSSI ................................ ...................... 8 1.4.3 Human activity analysis using Wi - Fi CSI ................................ ................................ ......... 8 1.5. Organ ization of Thesis ................................ ................................ ................................ .......... 9 2. Related Work ................................ ................................ ................................ .............................. 10 3. Contactless Indoor Activity Analysis using First - reflection Echolocation ................................ . 16 3.1. Introduction ................................ ................................ ................................ ......................... 16 3.2. Experimental System ................................ ................................ ................................ .......... 18 3.3. Activity Classification ................................ ................................ ................................ ......... 20 3.3.1. Distance Variation as the Classification Feature ................................ ............................ 21 3.3.2. Linear Regression Zero - crossing Quotient as the Second Feature ................................ . 23 3.4. Conclusions and Future Work ................................ ................................ ............................. 25 4. A Student Engagement Measurement System via Passive WiFi Monitoring ............................. 27 4.1. Intro duction ................................ ................................ ................................ ......................... 27 4.2. System Overview ................................ ................................ ................................ ................ 30 4.2.1. Dataset Description ................................ ................................ ................................ ......... 30 v 4.2.2 . Wireless Infrastructure ................................ ................................ ................................ .... 30 4.2.3. Data Collection, Filtering, and Processing ................................ ................................ ...... 31 4.2 .4. WiFi Monitor Placement and RSSI Cutoff Valu es ................................ ......................... 32 4.3. SEMINAR Set - Up Procedure ................................ ................................ ............................. 33 4.4. Dataset Analysis ................................ ................................ ................................ .................. 35 4.4.1. Validity of Collected Dataset ................................ ................................ .......................... 35 4.4.2. Weekly Patterns of Mobile Device Usage ................................ ................................ ...... 36 4. 4.3. Best - Case Scenario for Maximum Student Eng agement ................................ ................ 37 4.4.4. Spatial Analysis of Generated WiFi Traffic ................................ ................................ .... 38 4.5. Future Work ................................ ................................ ................................ ........................ 40 4.6. Conclusion ................................ ................................ ................................ .......................... 41 5 Human Activity Analysis by Exploiting WiFi Channel State Information ................................ 42 5.1 Introduction ................................ ................................ ................................ ............................. 42 5.2 System Design ................................ ................................ ................................ ........................ 44 5.2.1 Pus hup Data Classification ................................ ................................ ............................. 45 5.2.2 Pushup Data Repetition Counter ................................ ................................ ..................... 48 5.3 Evaluation ................................ ................................ ................................ ............................... 52 5.3.1 Pushup Data Classification ................................ ................................ ............................. 54 5.3.2 Pushup Repetition Counter ................................ ................................ ............................. 56 5.4 Limitation ................................ ................................ ................................ ................................ 58 5.5 Future Work and Discus sion ................................ ................................ ................................ ... 59 5.6 Conclusion ................................ ................................ ................................ .............................. 61 REFERENCES ................................ ................................ ................................ ................................ ........... 62 vi LIST OF TABLES Table 3 - 1: Pe rformance of two - feature classification ............. ..................................... .......................... 2 5 Table 5 - 1: Pushup Repetition Coun ter Performance ..................... ................. ............... ............. ......... .. 58 vii LIST OF FIGURES Fig ure 1 - 1 : Schematic representation of signal propa gation between Tx - Rx antennas ......... ..... .. ......... 7 Fig ure 3 - 1: Prototype EAD Device .................. .. .. ......... ............. .................................................... ....... 18 Fig ure 3 - 2 : System architecture for contactless activi ty monitoring ............... ........... ............. ... .. ......... 1 9 Fig ure 3 - 3: Raw tim e - series distanc e da ta from the prototype EAD .............. . . .. .... .................. ......... .... 2 1 Fig ure 3 - 4: Class distinctions using distance - COV .................. .. ... ............ ......... ........... ........... . ...... ..... 22 Figure 3 - 5: Sitting and standing classification accuracies .................. .. ........ .... . .. ... ..... .... . .. ... . .... ........... 22 Figure 3 - 6 : Using LRZQ for capturing frequency in distance signals ............ .......... . ... ... .... . ..... .. ......... 23 Figure 3 - 7: Two - fea ture scatterplots for all th ree activities ............ .......................................... . .. ......... 24 Figure 4 - 1: Different Components of the SEMINAR System Architecture ....... .................. .. ... .. ........ . . 3 1 Figure 4 - 2 : Monitor Placem ents for Data Collection .................. .. ..... ................................ ............... .... 32 Figure 4 - 3 : Sa mple Output of SEMINAR Set - Up Script ........ .............................................. ....... ......... 3 3 Figure 4 - 4 : Monitor Pl acements for Set - Up Procedure ......... ................................................. ..... . ......... 3 4 Figure 4 - 5 : Responsiveness of Monitor ing System ........... ..... ................................................. .... ......... 3 6 Figure 4 - 6 : Mo bile Device Usage for Different Weeks ............ ............................................... ... ......... 3 7 Figure 4 - 7 : Comparison of Traffic Patterns on Exam Days and Normal Fridays ....... ......... ........... ...... 3 8 Figure 4 - 8: Spatial Traffic Analysis During Explicit Browsing Sessions ....... .. ........... ....... ........... ...... 40 Figure 4 - 9: Pr oportion of WiFi Traffic Generated During Quiz Intervals ....... .. ........... ....... ........... ...... 40 viii Figure 5 - 1: System Overvie w .................. .. .... .................................................................. ................ ..... 45 Figure 5 - 2: Classification Breakdown ........ ............................................................... ... ... .......... .. ....... .. 46 Figure 5 - 3 : Heatmaps of Different Activities ...... .............................................. ............ ............ .. ......... 47 Figure 5 - 4 : Feature Separation Virtualizations .......... ........................................ ........... .... ........ .. ......... 48 Figure 5 - 5 : Pushup Exercise Sequence ......... .......................................................... ...... .... ......... .. ......... 4 9 Figure 5 - 6: Pushup Data Packet Correlation Matrices ...... .................................. ....... .... ............ .. ......... 50 Figure 5 - 7 : Push up Detection Counter Data Processing .................. .. ..... ............................ ............. .... 51 Figure 5 - 8 : Full Pushup Detection Counter Data Processing ......... ................... .............. ....... ... .. .. ......... 52 Figure 5 - 9 : Data Collection Environment and Device Placements ................ ........... ............... .. .. ......... 54 Figure 5 - 1 0 : Confusion Matrix fo r Pushup Data Classifier Pe rformance ........ ..... .. .... . ...... ............... .... 55 Fig ure 5 - 1 1 : Confusion Matrix for Pushup Classifier Performance without Multi - Class Outputs ... 5 6 Figure 5 - 1 2 : Classifier Performance .......... .......................................................... .............. ....... . .. ......... 5 6 Figure 5 - 1 3 : Multiple CSI Signatures .... .. ...... ............................................. ....................................... ... 5 9 Figure 5 - 14 : Mult i - Pe rson Wor k out Scenario .... .............................. . ................... ........ .............. .. ......... 6 0 1 1. Introduction I n recent years, Human Activity Recognition (HAR ) has become an essential paradigm in countless applications that enhance the health, safety and overall well - being of all humans. Human activ ities are es sentially a set of particular actions and movements in a given environment. A human activity recog niti on system aims to recognize these actions to provide activity specific information or gain contextual information about the surrounding enviro nment. Th is information can be used to analyze and understand human behavioral patterns. Traditionally, low - co st w earable sensors (e.g. accelerometers) fueled the development of sensor - based HAR systems [1 - 2] that utilize valuable sensor data to classify h uman acti vit ies. For example, wearable sensors are extremely useful in healthcare applications where the real - time sensor information alerts proper personnel of fallen elderly patients in distress. However, t he need for users to wear sensors for activity r ecognitio n i s inconvenient and impractical for long periods of time. This unavoidable limitation has dramatically increased the need for contactless human activity recognition systems [3 - 5 ] . Contactless HAR systems have increased the abilities, practicali ty, and c onv enience of sensor based HAR systems. It allows for unobtrusive measurement and classification of seden tary behaviors in the workplace such as time spent sitting at desk, to intense physical activi ties such as at home exercises. With unique syst em config ura tions and novel features extracted from collected sensor data, this work provides contributions to con tactless HAR methods and techniques. 1.1. Focus of Research This work presents an alternative to wearable sensors by classifying human activity wi th a fixed s ensor placement. A fixed sensor placement approach allows for uninterrupted data collection since the collection is not dependent on subjects wearing devices to initiate a data collection sess ion . Ultrasound sensors are low - cost sensors used fo r many ran gi ng and level detection applications, making it a great sensing modality for sensors with a fixed place ment. T his work utilizes ultrasound sensors for detecting numerous office s etting activities. A benefit of using a sound source for activity r ecognition i s the lack of interference from the ubiquitous RF devices. Despite RF rich office environments , ultras ound was used to confidently classify 2 human activities in office setting. On the other hand, the sensing range with ultrasound is limited due to the f ix ed directionality of the trans mitter. If a subject is not in the sensing range of the sensor then no dat a is collected . L imited sensing range can be a limiting factor in regard to human activity recognition . While activity recognition can be ach ieved with l ow cost ultrasound sensors, we show the beneficial impact of adopting radio frequency as a technologic al means of recognizing human activity. Specifically, WiFi signal analys i s enable s both macro and micro level activity classification. The ubi quity of W iF i infrastructures in home, university, work, and even outdoor enviro nments makes WiFi the most conveni ent technology to passively detect and classify human activity without the need of wearable sensors. 1.2. Contactless Sensing Technologies 1.2.1. Ultras ound Ultras o und frequency range begins at 20kHz, just above human audible sounds, and can extend past 10 MHz . An u ltraso und sensor uses one or multiple transducer s to send and receive ultrasonic pulses . The data received from the sensor generally correspon ds to an ob j to the sensor . High - frequency sound waves reflect from objects to produce distinct ech o patterns . When transmitting, the transducer converts the electrical energy into sound after which the sound wave is transmitted. U pon receiv ing t he ech o , the sound waves are converted into electrical energy which can be measured and analyzed . Often, ultr asonic sensors are placed into arrays to capture more information than a single sensor . Typical applications that utilize ultrasonic sensors i nclud e heal t hcare, home automation, energy efficient buildings etc. These low - power, low - cost sensors make ultraso und an ideal technology for recognizing various human activities. 1.2.2. Radio Frequency (RF) The frequencies of radio waves range from 3 kHz to 300 GHz. RF s i gnals can cover a large distance and can penetrate opaque objects like wall and h uman body. Many RF co mmunication systems that have been deployed in industry and in our daily life use different frequency ranges to communicate. The variety of RF technologi e s utilized for RF sensing applications such as Bluetooth [6] , Zigbee [7] , Radio F requency 3 n Device (RFID) [8] , and WiFi [9] , stem from the various communication ranges established within the radio frequency range. Bluetooth . Bluetoo th is used to transfer data over short distances. It is defined in IEEE 802.15.1 as a form of Wireless personal area networks (WPAN). The latest version of Bluetooth, i.e., Bluetooth Low Energy (BLE), can provide an improved data rate of 24Mbps and coverag e range of 70 - 100 meters with higher energy efficiency, as compared to older versions [6] . Du it is present in most modern wireless devices, however, the range of applications is typically limited to localization. Z igBee . Zigb ee is built upon the IEEE 802.15.4 standard that is concerned with the physical an d MAC layers for low cost, low data rate and energy efficient personal area networks [7] . Zigbee is heavily used in Wireless Sensor Networks (WSN). The Application Layer is r esponsible for distributed communication between nodes. The Network Layer in Zig be e is responsible for multi hop routing and network organization. ZigBee is used often for localization in sensor networks; however, a drawback is that it is not re adily avail able on majority of the wireless devices. Radio Frequency Identification Device (R FID) . RFID is a wireless data - capturing technique that utilizes RF waves to store and retrieve data between reader and tags that are generally attached to objects. RFID signa ls are affected by the surroundings so changes in the envir onment can be observed by monitoring by features of RFID signals. RFID signal features include RSSI and phase which can be obtained from a RFID reader [8] . There are two types of RFID sy stems. An a ctive RFID system depends on the internal power supply to r eflect a response to th e reader. Although longer ranges can be achieved, active RFID systems usually have a higher cost and larger form factor. Passive RFID tags draw much attention beca use of thei r smaller size and lower cost, and no need for power source s. A drawback of RFID i s the need for additional hardware. 4 WiFi . WiFi is perhaps the most popular technique used for RF sensing. Recent advances in wireless technology have found that t he WiFi sig nals are sensitive enough to capture environmental dynamics thus can be used for t he sensing purpose [9] . WiFi based sensing makes it possible to deploy sensing tasks on existing infrastructures, since WiFi access points are ubiquitous in typica l indoor se ttings. Additionally, WiFi is non - invasive sensing. The pres ence of humans and the ir related body movements will have considerable impact on surrounding wireless signals, thus human body movements involved in daily activities can be passively ca ptured and classified. Sensing tasks can be accomplished without user a wareness which provide s no discomfort. For the above reasons, we utilize WiFi, instead of other RF modalities, to achieve contactless human activity recognition. 1.3. Sensing Techniques 1.3.1. Fir st - Reflectio n Echolocation U ltrasonic sensors emit ultrasonic waves to measure distance by ke eping track of the time required to receive an echo of the transmitted pulse. Very little processing and computational overhead is required to calculate the distan ce data . The distance of the object that reflected the transmitted sound wave can be calculat ed with the following formula: where D is the distance, T is the time between the transmission and reception, and C is th e speed of sound constant . The value is multiplied by 1/2 since T accounts for the total round - trip time of the signal. 1.3.2. Received Si gnal Strength Indicator (RSSI) In RF wireless systems, RSSI is th e measured power of a received radio signal. It is implemented in IEEE 802.11 standards. For most widely used wireless techniques ranging from UWB, ZigBee, and WiFi to cellular networks, RSS I is easily accessible. RSSI indicates the path loss of a received wireless sig nal with respect to the distance of the transmitter [10] . The loss of signal strength th at occur bet ween transmitter ( 1 ) 5 and receiver is known as propagation path loss. Path loss is a m ajor factor when analyzing wireless communication systems. Various propagation models have been developed to predict the loss of signal strength between transmitte r and receiv er . A commonly used propagation model that predicts the path loss with respec t to distance is the logarithmic distance path loss model [11]. It is noted that the average received signal power decreases with the log of distance. In this work we obtain RSSI values of received radio packets where the RSSI is expressed using the log - di stan ce path loss model. According to the path loss model , the p ath loss between any transmitter and receiver can be expressed as: w here d is the distance, d 0 is th e reference point at 1m, and n is the path loss exponent [11]. The higher the RSSI values , th e complex indoor environments. RSSI not only varies over distance but also su ers from sh adow fading and tic li However, in typical indoor environments, wireless signals often propagate via multiple path s called multipath propagation. Wit h sever mult ipath propagation, RSSI may no longer decrease monotonically with propagation dis ta nce, thus limiting ranging accuracy. Multipath propagation can also lead to unpredictable minute even for a static link [ 11 ]. Since RSSI is a single - valued, it fails to convey kn owle dg e about multipath propagation, making it less robust and less reliable. 1.3.3. Channel State Information (CSI) In wireless communication, CSI is a representation of the channel pro perties of a communication link. As signals propagate through multiple paths, eac h of the paths leads to different amplitude attenuation, phase shift, and time delay. orthogonal frequency division multiplexing (OFDM) technology is a bandwidth - e fficient mul ti - subcarrier modulation scheme that combats multipath propagation, enabling sign als to be reliably received [ 12 ] . In OFDM, signals are transmitted over orthogonal frequencies, which ( 2 ) 6 are called subcarriers. Based on OFDM, CSI describes the chan nel properti es of a communication link and is able to distinguish multipath propagation a t th e subcarriers level. Channel measurement at the subcarrier is available on modern devices that adopt obey IEEE 802.11n/ac standard which permits multiple transmit and receive antennas for multiple - input multiple - output communication in wireless communi cati on. Therefore, CSI reveals fine - grained characteristics of wireless signals combining effect of time delay , amplitude attenuation, and phase shift of multiple path s on each co mmunication subcarrier. The WiFi signal can be modeled as Channel Impulse Res pons l l represent the amplitude and phase of the l - th multi - path component respectively [12]. l is the time delay, L indicates th e total number of multi - The time domain channel response h(t) can be derived by taki ng the Inverse Fourier Transform (IFFT) of the Channel Frequency Response (CFR). Every preamble of OFDM symbol s that is tr ansmitted can generate a channel matrix estimation from WiFi devices. This channe l matrix, otherwise known as, CSI is represented by: where both amplitude and phase information can be extracted [12]. The multipath effect, which is normally d etrimental f anal ysis of signal propagation behavior within a W iFi - covered area can identify and measure different types of disturbances. CSI represents the coefficient of a wireless channel a nd the CSI of every sub - carrier is a complex number . The CSI of a packet transmit ted with M transmitting antennas, N receiving antennas, 20MHz channel bandwidth, is a complex matrix of size M×N×56. If the bandwidth is 40MHz, then the size of CSI matrix bec omes M×N×114. Typical systems have multiple input multiple output (MIMO) antenna configuration. Figure 1 gives an overview of a n M × N MIMO system with M transmitting antennae and N ( 3 ) ( 4 ) 7 receiving antenna. Environmental changes and human body movements affect the C SI values of different links independently, but the effect on different subcarr iers of each link may be correlated. Authors in [ 13 ] developed the CSI tool that can extract the CS I from Wi - Fi implementing the IEEE 802.11n s tandard using a Network Interfac e Card (NIC) with Atheros chipset. 802.11a/g/n receiver s implements an OFDM syste m with 56 subcarriers for each receive antenna, therefore, fine grained changes in the wireless cha nnel can be observed. Figure 1 - 1 : Schematic representation of signal propa ga tion between Tx - Rx antennas 1.4. Research Objectives In this work , s everal re search objectives were met to produc e va luable contributions in human activity recognition. 1.4.1 Human a ctivity analysis using ultrasound A n ultrasound echolocation - based approach for human activity recognition in indoor settings was developed to differentiat e between typical workplace activities such as sitting, standing, and walking . The key novelty of the system is to perform activity analysis using time - series distance data est i mated through first - reflection echolocation. With the use of supervised machine learning mechanism s, extracted features were used to cla ssify the unique signatures of different activities, a novel contactless method to passively monitor human activity in o ffice setting was developed. Th e complete system is a practical human monitorin g device that can rep lace the traditionally used wearable devices. 8 1.4.2 Student e ngagement m easurement u sing Wi - F i RSSI Collective Wi - Fi RSSI measurements from classroom generated w eb traffic was leveraged to develop a passive monitoring system with the object ive of providing a contactless method to obtain classroom engagement levels. The system monitors wireless network tr affic on both 2.4GHz and 5GHz ISM bands in a university cla s s room setting to enable capture of spatiotemporal contextual knowledge of studen device usage. By capturing spatiotemporal usage of mobile devices during lectures, student engagement lev els can be estimated in real - time using the resulting time - s e ries data of generated web traffic intensity. xtended use of ubiquitous mobile devices during classroom lectures is unobtrusive and free from human involvement, c onsensus, or bias. Additionally, the design of the low - cost s ystem yields a feasible approach of automatic measurement of student engagement at very high frequency (e.g. lecture - to - lecture). Therefore, the developed system provides as an instructional tool to easily and frequently measure and analyze the effects o f various classroom dynamics in order to enhance instructor performance, increase student engagement, thus improving overall quality of education in university classroom settings. 1.4.3 Human activity an alysis using Wi - Fi CSI The channel state information of mul t iple Wi - Fi commu nication lin ks between two mobile devices was analyzed to recog nize human exercises, namely push - ups, and autonomously provide detailed repetition data to track volume and frequency of physical activity across time without the use o f weara b le sensors. The effect of periodic movement s that human bodies create on nearby wireless signals can be observed and characterized to obtain sufficient contextual knowledge regarding specific physical exercises . Since Wi - Fi CSI is obtained from signals mo d ulated using ODFM, subcarrier level analysis was conducted to identify fine - gra ined movements . Specifically, we show that the correlation of subcarrier amplitude variations between subsequent received packets facilitates the separat ion of exercise data fr o m non - exercise data . Various pre - processing techniques were used to prepare the obtained CSI data for correlation feature extraction , yield ing high machine learning classification results . Additionally, through spectrogram 9 ana lysis, the number of repetiti o ns of an exe rcise can be determined and after multiple activity intervals the s tored exercise data provides a historical physical activity level database for the user . More importantly, we aim to extend the current literature convention of restricted devi c e placement a nd orientation when developing a system to detect or differentiat e between various human exercises. By exploiting antenna diversity a s well as subcarrier level analysis, we demonstra t e that exercise data can be accurately classified and analy z ed with a n unconventional device placement and orientation . 1.5. Organization of Th e s is In Chapter 2, a thorough literature review o f human activity recognition work is presented. In addition , comparisons were made to highlight and differentiate the contribut i o ns of the proposed work from the contributions in the reviewed literature. Con tactless activity recognition via t he developed prototype hardware system , Echolocation Activity Detect or (EAD), that is capable of measuring object distance using ultrasound e c h olocation with first reflection is presented in Chapt er 3. The low - cost system , Student Engagement Measurement s and INstructor Assessments in Realtime (SEMINAR), is discussed in Chapter 4 where s everal drawbacks of current student engagement measurement m e thods are addressed by the proposed system. Chapter 5 emphasizes the motivatio n to design Wi - Fi based human activity recognition systems as Wi - Fi CSI data analysis enables classification capabilities that cannot be obtained using ultrasound. The develope d monitoring system can successfully classif y can pushup exercise data and count the number of repetitions . Chapter 6 provides a detailed summary and conclusion of the prese nt ed work. 10 2. Related Work W e review related activity recognition work of each pr opos ed scheme, respectively . Our proposed office activity classification system via echolocation is first compared with exi sting u ltrasound - based activity recognition approache s . Next , we distinguish our proposed Wi - Fi monitoring based approach for measuri ng s tudent engagement in university classroom from existing Wi - Fi monitoring works in university settings . Finally, we thoroughly review existing CSI - based exercise activity re co gnition approaches and outline limitations of current systems which motivate t he p roposed exercise recognition system . (i) Ultrasound Based Human A ctivity Recognition Approaches U ltrasonic echolocation is used in [14] for mapping a n avigation applications. I n this approach, t he human subject that is navigating is required to wear the developed echolocation device, so that audible cues translated from reflected ultrasound signal can be acted upon for effective navigation. While this wo huma n ac tivity is not explicitly classified in this work. Additionally, the system is a wearable device, and therefore suffers from the same limitations as sensor - based activity re co gnition systems. Echolocation is used for a utomatic fall detection in [15] and bre athing monitoring in [16] . Both works use full - reflected signal analysis to detect movements of their respective activities, movements of a human fall and chest movements c au sed by breathing. A drawback of these works is the hardware computational comp lexi ties of the signal analysis . Our approach which only uses first - reflection a nalysis saves on computational resources as opposed to full - reflected signal analysis. The appro ac hes reported in [1 7 ,1 8 ,1 9 ] target contactless activity analysis using ultrasou nd e cholocation and t he applications in those papers are similar to our target a pplication . H owever, all of those approaches also rely on full - reflected signal analysis . By con ve rting the first ultrasound reflection to a signal that provides the distance b etwe en the echolocation device and the nearest obstructing object , a low - complex ity post - processing system for activity classification is presented . 11 (ii) Wi - Fi Monitoring Based A ct iv ity Analysis in Classroom Settings Researchers in [20] developed EDUM, an educ atio n measurement system that measures class punctuality (if students were on time, late, or attended a class at all) and lecture attractiveness. EDUM uses device - specific pers on al data including MAC address of mobile phone, student identification number, and course schedule of specific students. Class attendance is determined by recording the RSSI of an ro om. In addition, EDUM requires installation of two mob ile apps on phones of st uden ts. Using the installed apps, lecture with respect to the length of t he class lecture. This work is limited in that it requ ires students to volunte er i n participating via installing and using a third - party App. Students may not always be open to that, especially if they know that the App will be used for tracking their de vi ce usage. The work in [ 2 1] addressed the problem of r estricting network acces s in side specific university classrooms. A two - part system containing an Ethernet bridge with a web - based control panel and a WiFi monitoring station was used. The Ethernet bri dg ith the goal of using a web - based control panel to modify traffic flow rules that will ultimately limit which internal network traffic is able to leave the campus network. A WiFi monitor is placed ins id e a classroom to monitor probe packets that will deter mine which MAC addresses are inside the classroom. Over Ethernet, the monitor sends a list of current MAC addresses to a database containing a list of classrooms and MAC addresses that are inside of t ho se classrooms. Given that the instructor wants to res trict network access, pa cket controller. This work does not attempt to gain behavioral knowledge of stude nt s rather it aims to restrict Internet access from devi ces in a room by using a WiF i monitor to develop a list of nearby MAC addresses whose Internet access should be restricted. This can cause trouble for nearby devices that broadcast a probe packet that i s seen by the monitor and is mistaken to be inside the classroom. 12 The paper in [22 ] investigated wireless traffic patterns in a university library and auditorium via WiFi monitoring. Several monitors are deployed throughout both locations with the primar y purpose of capturing individual device MAC addresses o btained from periodic pr obe requests. As a result, researchers are able to determine occupancy levels in each of the buildings for different times of the day as well as provide information regarding t he average stay duration of the devices. Similarly, [23] performs end user profi ling in university environment. One dataset was collected from monitoring inside a classroom and another from the research lab of the authors. Regarding user profiling, authors p roposed a set of features to be extracted from the cap ture time of probe reque st f rames to cluster users in groups (e.g. lab members or visitors) based on their dwell time and presence. Using the dataset collected in a classroom, authors show that probe re quests can be used to distinguish between smartphone a nd laptop. A tool aimed at is presented in [24] . This system requires knowledge of MAC addresses for all studen ts enrolled in the class. Using Wi - Fi monitoring, the au thors implement a locali zati on technique and map the MAC address of a located device to a student currently enrolled in the course. Information regarding which students are present is then provided to t he instructor. Our proposed system differs from the above works in the follow ing aspects. First, and most important, the above works do not attempt to gauge student engagement. [20] attempts to measure lecture attractiveness by providing a ratio of acti ve screen time to the duration of the lecture, however, this is ineffective when stu dents view lecture slides from their devices. Secondly, unlike many of the above approaches, SEMINAR requires no student participation such as mobile App installation, whic h makes the SEMINAR approach more pra ctical and feasible than many of the approa ches discussed above. In addition continuous monitoring from human observe rs , thus this approach is not vulnera ble to students being able to intentionally adj ust their classroom behavior in order to thwart any measurement efforts. Third, many of the approaches discussed above require tracking device MAC address level information t hat is mapped to 13 student Ids, thus raising privacy concerns. The SEMINAR appro ach avoids such information collection and mapping to student IDs. Fourth, unlike some of the approaches above, the measurement approach in SEMINAR is agnostic to the underlyin g encryption used by the mobile devic es. This is achieved by concentrating mainl y on aggregated traffic volume, and not device - specific traffic for which management frames need to be probed. Encryption can block the visibility of the management frames such a s the probe frames, thus limiting t he effectiveness of gathering device - specif ic i nformation. In SEMINAR, traffic volume can be measured even when the lower layer management information is blocked using various encryption methods. Finally, the work prese nt ed in this chapter specifically aims at gathering knowledge of device usage rath er t han attempting to detect the presence of or restrict the network access of a device in a specific location, as targeted by some of the above approaches. These differences c le arly distinguish SEMINAR from curre nt relevant literature that employ Wi - Fi mo nito ring in university settings to gain contextual information. (iii) Contactless Exercise A ctivity Recognition Contactless physical fitness and exercise monitoring has been extensiv el y studied literature. In [ 2 5 ], a ccelerometer sensors are embedded in to glove t o re cognize and track various free - weight exercises conducted by human subject . While high activity recognition accuracy is obtained, the requirement of wearing de dicated senso rs , especially for exercise activity recognition, may be unsuitable . A passive R FID based free - weight exercise recognition system in [2 6 ] attaches passive RFID tags to the training devices, i.e., dumbbell in this work, an d leverages the backscattered signa l for activity reco gnition . While RFID tags are not wearable sensors, this is a sens or - based method that requires many sensors for widescale implementation thus, practical implementation is limited. [2 7 ] aims to help the user to achieve effective workout a nd prevent injury b y dynamically depicting the short - term and long - term picture of a workout based on various sensors in common mobile devices. The above works achieve the goal of contactless exercise monitoring; however, wearable sensor - based imple me ntations have a s ignificant drawback . Despite its small size and light weight, sen sor - based systems require the user to wear the device 14 or keep it within close proximity for detection . The need to wear a sensing device to monitor physical exercises is el im inated with Wi - Fi based activity recognition, such as the proposed system. Ea rlie r Wi - Fi based contactless activity recognition systems utilize d RSSI as the sensing measurement. [ 2 8 ] propose d the concept of device - fre e passive localization using RSS I an d present a passive radio map construction to enabled device location tracking. Behi nd the wall localizing and tracking and localizing of a target using a statistical model of RSSI variance was proposed in [ 2 9 ]. [30 ] and [31] employed RSSI for gesture reco gn ition and intruder detection, respectively. Since RSSI is a very simple metric and does not require any special hardware changes u sing RSSI for human activity recognition is very easy , but RSSI suffers from severe multipath fading, distortions and instab il ity in complex environment s. Regarding the granularity of information from the obt ained data, RSSI is a coarse - grained information and it does not leverage the subcarriers of an OFDM channel like CSI. Fine - grained information can be obtained from advance d approaches that employ CSI to passively detect human activity since collected data of CSI is richer than that of RSSI . Activity recognition via CSI as the sensing measurement enables recognition of more specific activities both fine and coarse grained s uc h as smoking [ 32 ], breathing [ 33 ], and various gesture s [ 34 ] . K eystrokes from a co ntinuously typed sentence can be identified with high accuracy in [ 35 ] . T he number of people in a crowd can be determined using CSI values by treating the human subjects re fl ecting the Wi - Fi signals as virtual antennas in [3 6]. [3 7 - 39 ] all proposed a p assi ve human detection scheme which exploits multi - path variations for detecting human presence in an indoor environment using CSI . While there is extensive work done in activ it y recognition using CSI as a whole , the proposed work in this thesis aims dete ct p hysical exercise using CSI, avoiding any wearable device. Current works that also aim to achieve contactless detect ion or monitor ing of physical exercise s are [40 - 42] . [ 40 ] p ropose d a CSI - based green system for exercise activity recognition and quality eva luation . A novel method is proposed to use the complete CSI - waveform shape as a feature to successfully d etect the starts and ends of each action . Although this work relies o n the training data and selected features, the systems suf fers when the 15 deploy ment environment is not static. T he Fresnel Zone di f f raction model is leveraged in [ 41 ] to understand the principle behind Wi - Fi sensing effects and can detect various exercise a ctivities. While this can accurately distinguish between different exercises t he p lacement and orientation of the Tx and Rx devic es are constrained so the potential for practical system deployment in real world is decreased. [ 42 ] developed a system capab le of differentiating users that execute the exercise. The system uses a Deep N u eral N etwork based model to perform fine - grained workout interpretation and to provid e smart workout assessment. This work also implements a system whose placement and orientat io n of the Tx and Rx devices are constrained, which motivat es the proposed work. T h e human activity recognition sys tem that we propose sets us apart from current literature that uses CSI for exercise activity monitoring because we aim to lessen the placem en t constraints of the receiving ntio n on Tx - Rx placement and orienta tion. Currently, existing approaches place the CSI transmitting and receiving devices 3 - 5 meters apart where the respective height and orien ta tion of each CSI device is equal. To increase system flexibility and practical ity our work decouples the heigh t , o rientation, and physical placement of the receiving device from the CSI transmitting device by taking advantage of the fined - grained context ua l information embedded in CSI data of subsequent packets at the receiver. This all ows the transmitter (e.g. Access to accurately differentiate exercise data from non - exercise data with a Tx - Rx pl acem ent consisting of different devi ce height and orientation. Additionally, with advanced signal de - noising methods and novel feature extraction, we prove that exercise statis ti cs such as number of repetitions can still be determined with a non - standard d evic e placement. 16 3. Contact less Indoor Activity Analysis using First - reflection Echolocation T his chapter presents an ultrasound echolocation - based approach for human activity re cognition in indoor settings. The key novelty of the proposed approach is to p erfo rm activity analysis using distance estimated through first - reflection echolocation where the distance to the nearest obstructing object is computed using the first reflect ed ultrasound signal. All subsequent reflected signal components from other dist ant objects are ignored. This leads to an extremely simple signal (i.e., time - series distance data) analysis approach with very low computational complexity. Especially so, whe n compared with the existing approaches in literature in which full reflected si gnal analysis, often with Doppler Shift computation, is performed for activity classification . It is demonstrated that for the goal of isolating workplace sedentary behavior, t he proposed approach can differentiate between sitting, standing, and walking (i .e., in - office pacing) with more than 80% accuracy. This was validated with different classif iers applied on data collected from multiple subjects in multiple sessions. Recorde d video was used as the ground - truth for training the classifiers. 3.1. Introduction In recent years, human activity monitoring and analysis have gathered significant attent ion [ 43 ] from medical researchers as well as health - conscious consumers. Majority of ac tiv ity monitoring devices use multi - axes accelerometers and gyroscopes [ 44 ] to d etec t body movements in order to detect human activities with various granularities. Huma n activity is being considered as one of the primary actionable indices within the evol vin g framework of quantified self [ 44 ]. Research indicates [ 45 ] that presenting one' s activity data in a meaningful manner can improve one's overall lifestyle both in te rms of dietary and physical activity habits, thus alleviating heart disease, diabetes, dep ression, and other health risks. Recent proliferation of wearable activity mo nito ring devices from Apple, Samsung, Sony, Fitbit, Garmin, and other vendors indicate th e exploding consumer interest for activity monitoring. While the wearable devices off er ubiquity and continuity of monitoring, a major limitation is that a consumer does have to actually wear them for continuous monitoring in both indoor and outdoor se ttings. 17 Many recent studies suggest [ 46 ] that in addition to the disadvantages of losing and having to remember to regularly wear, the interest in such devices tend to w ane within a fairly short period. This is more so for healthy gadget - enthusiasts, for w hom the interests in such devices typically do not last for more than three to six months [ 4 7 ]. Although it is gradually becoming less of an issue, the lack of cosmetic appe al of such devices in certain situations can deter people from using them. One way to eliminate the need for wearing devices is to monitor activity by analyzing video foot age from strategically placed cameras. However, video would work only in indoor sett ings, thus operating with much less ubiquity compared to the wearable solutions. A bigger issue with video is its intrusive nature leading to privacy concerns. Additionally , a ny real - time feedback or intervention process based on video - monitored activi ties requires complex image processing which is not the case for wearable device - based solutions. Availability of proper lighting conditions can also pose a challenge for such ima ge processing tasks. In this chapter, we propose a privacy - preserving activit y mo nitoring using ultrasound SONAR, which is used by many mammals includ ing bats, dolphins, and whales. The key idea is to emit ultrasound pulses towards an individual and to ana lyze the reflected sound signal for estimating the current activity of the in divi dual. This, however, requires an ultrasound transceiver unit to be pl aced by the subject individual, thus limiting the ubiquity of the approach mainly to applications in in doo r settings. Its utility is also limited when multiple subjects are simultaneo usly visible by the SONAR transceiver. In spite of such constraints, th is technique can be leveraged in many niche applications including indoor monitoring: ( a) of an individu al in her/his office/cubicle where a significant number of the waking hours are spen t, ( b) in living room, ( c) an elderly person staying home alone, and ( d) sleep analysis. In many of these situations, having a continuously running SONAR unit in one or mul tip le places within a home/office can offer a more practical and reliable soluti on c ompared to the ones that mandate the monitored subject to wear a mon itoring device without fail. 18 Specific contributions of the chapter are as follows. First, a prototype har dwa re system that is capable of measuring object distance using ultrasound echol ocat ion with first reflection was designed. Using only the first reflect ion as opposed to full reflection analysis, as done in many approaches in the literature, offers an acti vit y analysis approach with ultra - low computational complexity. Second, a set of det ailed experiments, involving work - place activities, for multiple sub jects were conducted for collecting echolocation data and ground - truth video data. Finally, multiple fea tur es were designed for using the echolocation data in order to perform contact - less activity classification. 3.2. Experimental System All the experime nts and validation results presented in this work are targeted towards monitoring an individual in the workp lace . Fig 3 - 1 shows the prototype Echolocation - based Activity Detector (EAD), wh ich is placed on a desk by the office occupant. The EAD unit contai ns a microprocessor (ATmega328P on Arduino Uno R2 platform [ 48 ]) that collects data from an array of ultrason ic e cholocation sensors ( HRLV - MaxSonar EZ1 Ultrasonic Sensor [ 49 ]). The sensor d ata is then post - processed for classification of the activities. Figure 3 - 1: Prototype EAD Device 19 The sensor HRLV - MaxSonar uses 42 KHz ultrasound pulses and computes object d ista nces with millimeter resolution based on the first major reflected signal. A s re ported in its spec sheet, reliable distance measurement can be done for objects placed from 11 i nches up to 170 inches. It should be noted that these simple sensors with sm all form - factor (approximately 0.5in x 0.5in x 0.5in) do not full - reflected sign al a nalysis for detailed terrain recovery. Instead, they only measure the timing of the signal trans mission and that of the first reflected signal in order to compute the dista nce of the nearest obstructing object. All subsequent analysis for activity clas sifi cation is performed on time - series distance data. Note that although the prototype contains an a rray of four sensors, the results in this work are based on the data obtain ed f rom a single sensor. Figure 3 - 2 : System architecture for contactless activi ty m onitoring We target the problem of classifying three typical work - place activities, namely, si t, stand, and walk (i.e., in - office pacing). Studies have shown [ 50 ] that the se a ctivities can often reliably indicate sedentary behavior, thus able to predi ct v arious health outcomes. Data was collected from six subject individuals. For each person, each of the three target activities (i.e., sit, walk, and stand) was per formed for six different sessions, each lasting for around 30 seconds. During a session, the EAD unit was placed at an approximate 20 - series dis tanc e data between the EAD and the subject was collected during the entire sessi on. The hypothesis is that the measured distance and its variation can provide unique signatures for each of the targeted activities . Video recording was done for cap turing the act ual activity states, which are then used as the ground truth during machine - assi sted activity classification. Note that the EAD unit and the video recorder were kept in data collection mode even when the subjects walked out of the office. Thi s allowed sens or - based office occupancy detection. Figure 3 - 2 depicts the overall system co mpon ents to show how the EAD is placed by an office occupant, such that the sensors generally face the occupant. It collects first - reflection ultrasound signal to be able to co mput e distance of human - objects located from approximately 11 inches up to 170 i nche s with an EAD server through WiFi link. Activity clas sification sof tware : The left part of Figure 3 - 2 shows the classification software in the EA D se rver. It has an offline training component in which a classifier is trained for activity classification using sensor data and the ground truth obtained from time - stamped vi deo footage. The trained classification model is then used during real - time clas sifi cation. Using live data stream from the EAD, activities are classified as sit, stand, or walk, and fed into a sedentary behavior parser algorithm, which extracts sedentary beha vior statistics. 3.3. Activity Classification The key classification steps in t his section are: ( 1 ) feature design and extraction from time - series distance data, ( 2) classifier training, and ( 3) performance validation. 21 Figure 3 - 3: Raw time - series distanc e da ta from the prototype EAD Sample time - series distance data for two most com mon workplace activities, namely, sit and stand, are shown in Figure 3 - 3 for one female and one male subject. As can be seen, between sit and stand, the former consistently produc es h igher distance variations. The reason is that higher lower - body stability d urin g the sitting posture allows one to sway the upper - torso more ofte n than in standing posture when the overall stability is less. Since the sensor was targeted towards the su bjec - lobe width (i.e., at the subject - EAD dista nce of 36 inches to 60 inches) is approximately 50 inches, the sensor was able to capture the sway of the upper torso in terms of the distance variations as shown in Figure 3 - 3 . I t wa s consistently observed that walking produces much higher distance variatio ns w hen compared to sit and nce from the sensor varies significantly when she/he walks away or towards the EAD device. 3.3.1. Distance Variat ion as the Classification Feature Based on the above observation of distance v aria bility for different activities, we used Coefficient of Variatio n (COV) of the measured distance as the classification feature. COVs of the time series distance data are com pute d for each 30 sec. session, and the corresponding actual activity (i.e., th e gr ound truth) for the corresponding window is marked from the vide o recording. 22 Figure 3 - 4: Class distinctions using distance - COV The left panel in Figure 3 - 4 depicts the COVs for all three activities. Each point in the graph represents the COV of the di stan ce values measured over a 30 sec period. The COV is computed over 300 samples collected at the rate of 10Hz for 30 seconds. Observe that the high distance variability for wa lkin g clearly differentiates the points for walking from those for sitting and stan ding, thus indicating the suitability of distance - COV as a classification feature for walking. For instance, a threshold COV of around 0.3 is sufficient to separate out the walk ing events with almost no loss of accuracy. Between sitting and standing, h owev er, the COV demonstrates significant overlapping, which is more apparent in the zoomed - in version of the graph, as shown in the right panel in Figure 3 - 3 . Su ch overlapping wou ld l ead to loss of classification accuracy while separating standing and sittin g. Figure 3 - 5 shows how, after the walking events are separated out, those classification accuracies individually change for different thresholds chosen for the distance - COV. Figure 3 - 5: Sitting and standing classification accuracies 23 From these results, it can be concluded that using distance - COV as the single classification feature may not always be sufficient for simultaneous accurate classification of the sitting and standing acti vities. This leads to the two - feature classification solution as presented belo w. 3.3.2. L inear Regression Zero - crossing Quotient as the Second Feature The sample distance data in Figure 3 - 3 demonstrates that even though its variabili ty during standing and s itt ing are not significantly different, sitting produces a higher frequency sig nal. This general pattern was observed in all the data collected for all subjects. This observation led to the following addition of a frequency domain f eature for improved clas sif ication. Figure 3 - 6 : Using LRZQ for capturing frequency in distance signals Lin ear Regression Zero - crossing Quotient (LRZQ) is used for capturing the frequency of variability as follows. First, as shown in Figure 3 - 6, linear regression of the distance va lue s is computed. Then the count of how many times a signal touched or crossed this line was recorded. This count, termed as the LRZQ, indicates the general frequency content of the signal in question. For example, it is evident from Figure 3 - 6 that the LRZQ fo r sitting is significantly larger than that for standing. This was generally obs erved for all collected data for all subjects. As done for the COV feature, the LRZQ values are computed for each 30 sec period of the collected data. Note that LRZQ, instea d o f Fast Fourier Transform (FFT) based spectral density computation, was used for maintaining computational simplicity and the subsequent ease of run - time classification abilities within an embedded setting. 24 Figure 3 - 7: Two - feature scatterplots for all th ree activities The left panel in Figure 3 - 7 shows the scatterplot of COV and LRZ Q fo r all collected data for all three activities for all subjects. Each point on the graph indicates the COV and LRZQ computed over one session (i.e., 30 sec) worth of distance da ta. Observe that for walking, while the COV is higher than the other two act ivit ies, its LRZQ is quite small, because the periodicity in movement is of the order of walking strides, which is high compared to t hose for sitting and standing. This causes a ve ry distinct point cluster for walking. Consequently, similar to the one - feat ure classification case, walking can be easily classified against the other two activities. The right panel in Figure 3 - 7 shows the zoomed - in scatter plots only for sitting and s tan ding. It should be noted that in this two - feature scenario the clusters for thes e two activities are much more separable compared to the COV - only scenario in Figure 3 - 4. With COV and LRZQ as the features. Thr ee different classifiers, namely, Logistics, Na ïve Bayes, and Sequential Minimal Optimizer (SMO) were tried with 90 - 10 validat ion method. In this method, 90% of randomly chosen data points are used for supervised training of a classifier and the remaining 10% of the data are used for validation purpose s. 25 Table. 3 - 1 : Performance of two - feature classification Table 3 - 1 summarize s th e classification performance in terms of precision and recall . Precision (also termed as positive predictive value) for a class (e.g., sitting) represents the percentage of all classified activities that are actually sitting. Recall (also termed as sen siti vity) represents the percentage of actual sitting activities that are classified as sitting. Table 3 - 1 shows excellent classification performance (i.e., more than 80% for all cla sses) with all three classifiers. 3.4. Conclusions and Future Work We proposed a nd implemented a novel contact - less human activity monitoring method that uses ultrasound - based echolocation utilizing only the first reflection , indicating the distance to th e ne arest obstructing object. The key idea is to use a SONAR - based E cholocation b ased Activity Detector (EAD) distance of the closest point on the tor so. The time - series distance data contains signatures of differe nt activiti es , which are detected by classifying those signatures using supervised machine learning mechanisms. Using a prototype EAD, developed in our laboratory, we demonstrate the effec tive ness of the approach for a specific application of work - place ac tivity anal ys is. Three typical work - place activities, namely, sit, stand, and walk (i.e., in - office pacing ) are detected using the echolocation signal. It was shown that using a two - featur e cl assification approach, it is possible to identify such activitie s with more t han 80% accuracy over many subjects and activity sessions. 26 Future work on this topic includes: a) experimenting with run - time classification with the goal of providing real - time feedback, b) exploring Fast Fourier Transform (FFT) spectral co mponents as c lassification features, c) using a sensor array for improved spatial resolution of the distance data, d) scaling up the system for more activities applicable for a larger set of a pplications, and e) characterization of the system under ultraso nic ambient n oise, often produced by motors in home/office appliances. In the next chapter, we transition from ultrasound to WiFi as the sensing measurement to attain contactless human ac tivi ty classification. 27 4. A Student Engagement Measurement System via Passive Wi Fi Monitoring A contactless method to obtain collective classroom engagement levels is presented in this chapter. The ubiquity of mobile devices and wireless infrastructures in univ ersity classrooms have aided student learning experience in recent years. A mo ng other things, using such technology, the students are able to follow along lecture slides and search through key concepts fro m the Internet in real time in order to better comp rehend the instruction material . While aiding learning experiences, these d ev ices can also be used for non - academic Internet browsing, which is a major source of distraction , and can negatively impact the overall quality of education . This work attempt s to measure spatiotemporal usage of mobile devices during lectures by passivel y monitoring WiFi traffic generated inside class rooms. Such measured data can be used to determine student engagement in real - time . Using temporal patterns of classroom generate d Wi Fi traffic, the proposed system can generate automatic measurements of stud en t engagement in real time, and for longer durations in various time scales. The work proposed in this chapter demonstrates the f easibility of a practical WiFi traffic monitori ng s ystem that implements a contactless approach to monitor student activity wi th regard to use of mobile device within the classroom. Ultimately, the system can be used as an instructional tool to measure and improve student engagement in university class room settings . 4.1. Introduction Wireless infrastructures in universities have bec om e ubiquitous over the past decade. The advancement s and ubiquity of such wireless infrastructures have significantly fuel ed the increase in number of mobile devices. A 2017 st udy [ 51 ] shows 97% of college students own smartphones and 95% of college students own lapto ps. M any students us e such mobile devices to take notes and/ or follow along lec tu re slides . While these mobile devices can aid classroom learning, they can also contribute to stud ent distractions and negative ly impac t the quality of education . The objectiv e of this work is to develop a system for estimating such distractions in an un ob trusive manner. Student engagement in higher education is heavily researched in the discipline of educational psychology [ 5 2 ] and it is proven that higher levels of stude nt engagement has a strong link to greater academic success 28 and achievements [ 53 ] . Work pertaining to student engagement can be grouped into the following categories : defining and understanding student engagement [ 54 ] , understanding how student engagement affects quality of education [ 55 ] , and methods of measuring student en g agement [ 56 ] . We attempt to measure s tudent engagement using the propose d system, S tudent E ngagement M easurements and I N structor A ssessment s in R ealtime (SEMINAR) . SEMINAR is a passive network traffic monitoring system that provides spatiotemporal knowledge of mobile device usage inside a c lassroom, which can provide a reliable and r ea l - time measurement of student engagement during a classroom lecture. Definition of Student Engagement . Student engagement has three interrelated aspects: behavioral, cognitive, an d emotional. This work focuses on the behavioral aspect which encompasses p os itive student behaviors including effort, participation, attendance, and compliance with other classroom norms [ 57 ] . Based on our hypo thesis of a correlation between in - class mob il e device usage and student distractions, for this cha pter , we define student engagement as the absence of distraction induced by extended non - academic use of mobile devices. The ke y idea of the proposed framewor k is to passively monitor in - class WiFi netw or k traffic so that mobile device usage inside a classroom can be estimated . In literature, applications of WiFi monitoring include localization [ 58 ] , crowd sensing [ 59 ] , and facility management [ 60 ] . SEMINAR makes use of data frames captured, contrary t o the vast majority of WiFi monitoring wor ks that only make use of management frames. Data collected by the system is collective, and so, privacy related issues tied to device - specific data collection are avoided. The complete construct and implementat ion of our low - cost, classroom specific, and pa ssive monitoring system are discussed in this work. Obtaining knowledge regarding student engagement is extremely beneficial as it provides instructors and university administrators the ability to monitor and im prove courses with the ultimate goal of im pr oving the quality of education. A drawback of current state of the art observational student engagement measurements is that observers generally observe a sample of students in a class rather than the whole cl ass . In addition, this method requires a cons en sus of student behavior among all observers. The most significant 29 drawback of observational methods, as well as other methods (e.g. student/teacher self - assessments, questionnaires, etc.) is that engagement me asu rements are concluded the moment observers d iscontinue their observations or questionnaires are no longer distributed. The aforementioned drawbacks raise a question that motivates our current work. After obtaining results of classroom engagement via con ven tional measurement methods, h ow can i nstru ct ors be sure that changes in lecture dynamics (e.g. teaching style, presentation of material, etc.) to improve student engagement continue to make positive impacts ? In other words, it is difficult to continuall y g auge the retention of corrective instructi on al measures using the traditional methods. This is difficult due to the high operational costs such as constant human presence for student observations, and it is often unreasonable to believe that current me asu rement methods can be consistently conduct ed - to - day, month - to - month, or even semester - to - semester basis. However, engagement measurements at high frequency may provide a better understan din g as to how methods and practice of teachi ng are linked to student engagement. The proposed system, SEMINAR, addresses many of the challenges and drawbacks of current student engagement monitoring measures. Specifically, it can provide unobtrusive t empor al and spatial engagement measurements in re al - time to instructors. In this chapter we present and showcase the ability of the system to measure WiFi traffic volume that can be used to provide measurements of student engagement inside a classroom. We al so de monstrate and highlight the feasibility of a utomatic measurement of engagement at very high frequency (e.g. lecture - to - lecture) without any human involvement. Specific contributions of the cha pter are as follows. First, it presents the framework of a sy stem that passively monitors wireless netw or k traffic on both 2.4GHz and 5GHz ISM bands in a university classroom Second, it d emonstrate s how extended u se of ubiquitous mobile devices can provide a m ea sure of student engagement that is free from human involvement, consensus, or bias. Finally, the chapter s howcase s the feasibility of the proposed system to be effectively used in a slew of upper level applica tions that will be developed using the develope d measurement framework . 30 The rest of the chapter is organized as follows. Section 4.2 presents the overall system architecture and Section 4 .3 outlines the system set - up procedure. Extensive analyses of collected data is presented in Section 4.4 and a di sc ussion of our current and future work is presented in Section 4.5 . The chapter concludes with Section 4.6 which provides the conclusions of this work. 4.2. System Overview In this section we discuss our current datase t and how data is collected. We also provi de details regarding our system design and set - up. 4.2.1. Data set Description The data collected in this work was collected by attending multiple lectures of a second - year e ngineering course throughout a 14 - week semest e r. It should be noted that in this partic ul ar course, there were no online notes or class material provided. The instructor generated all class notes in real time; as a result, there was no immediate need to ac cess the Internet during the lectures. Collec t ed data is used to demonstrate the abilit ie s and feasibility of the presented SEMINAR system. Also, to be noted that the main goal is to capture a collective view of the classroom traffic, and therefore, no a p riori student device - specific information is c ollected. 4.2.2. Wireless Infrastructure The W iF i infrastructure in university buildings on our campus consist of multiple IEEE 802.11 access points (AP) manufactured by Aruba or Cisco that are distributed throughout the corridors of each building. A wireless d istribution system (WDS) is implemented, ef f ectively broadcasting 2 SSIDs primarily used by occupants to gain Internet access anywhere on campus, MSUnet and MSUnet Guest. Each AP may broadcast multiple SSIDs covering both 2.4GHz and 5GHz ISM bands which u t ilizes many of the available WiFi channel s. Since there are no APs located inside classrooms, the current wireless infrastructure makes it difficult to monitor the amount of traffic in one specific room. Furthermore, gaining access to view data from speci f ic APs that are closest to the target cla ss r oom would result in unusable information since data to and from each AP can be from any of the surrounding rooms. 31 Figure 4 - 1: Different Components of the SEMINAR System Architecture 4.2.3. Data Collection , Filtering, and Processing As depicted in Figure 4 - 1a, d ata is collected in a univ ersity classroom. Our monitoring device, sho wn in Figure 4 - 1b, is a Toshiba laptop with a Linux operating system (OS) installed. It should be noted that any laptop with a standard network interface card installed is able to enter mo nitor mode; Linux OS makes it easier to control the network interfaces . The monitor is equipped with 2 WiFi interfaces in monitor mode to allow sufficient data capture in both 2.4 and 5 GHz frequency ranges. Chan nels used in 2.4 GHz band are 1,6, and 11, s o one WiFi interface monit ors these channels at a rate of 3 channels p er second. After initial analyses of wireless infrastructure, it is observed that there are 4 channels in the 5 GHz band that are primarily us ed by devices in the classroom. As a resul t, the second interface is u sed to monitor channels 36, 48, 60, and 161 at a rate of 4 channels per second. The collected data is wirelessly sent to a storage database where filtering and processing occurs. We mea sure the volume of uplink data frames gene ra ted by devices inside a cl assroom, so we filter out management and con trol packets. In addition, we filter out downlink data frames since these frames are transmitted by APs located outside the classroom. Finally , to extract only data generated by device s inside a classroom, we fi lter out frames with RSSI value outside of our determined RSSI cutoff value as discussed in the next subsection. After filtering out all unneeded data, the remaining data is processed an d wirelessly sent to a viewing platform (i .e . tablets, iPads, or lapt ops) owned by instructor. This enables real - time display of cur rent 32 measurements. Currently displayed is raw traffic intensity measurements; however, the next step is to correlate thes e measurements to a level of engagement an d display engagement levels as well. 4.2.4. WiFi Monitor Placement and RSSI Cutoff Values For o ur system to provide reliable spatiotemporal Wifi traffic analyses, monitor placement is a crucial step in the data collectio n. A WiFi monitor is placed in the center of a classroom as in Figure 4 - 2a to capture collective WiFi usage in the classroom over tim e. For more granular information, it can be determined how much WiFi traffic each section of a classroom (front, back, left, or right) generate s by also placing monito rs in locations shown in Figure 4 - 2b. These placement locations are further referred to as edge placements. Depending on the size of a classroom, the effective monitoring range of a monitor may extend past the walls of a room; in this case it is imperative t o have a RSSI cutoff value determined. This is visualized in Figure 4 - 2. For brevity, we r efer to the RSSI Cutoff Value as RCV. When a monitor is placed in the center of a room as in Fig 4 - 2a, only one RCV is needed . Monitors at edge placements (Fig 4 - 2b) r eq uire two RCVs, one for the far end of the room and another that is used to approximate the midway point of the room. Determining an RCV to approximate the middle of the room effectively allows measurements to be divided between fr ont and back of classroo m. Similarly, the midpoint RCV allows the room to be divided into left and right sections . Figure 4 - 2 : Monitor Placements for Data Collection (a) Front Mon itor Placement (b) Edge Monitor Placement s 33 4.3. SEMINAR Set - Up Procedure Before SEMINAR can be effectively used, an in itial set - up procedure must be executed to d etermine the needed RC Vs. Since the topology and size of a classroom are two features that effect the RCV, this set - up procedure must be executed anytime the system is deployed in a new classroom. To highlight th e feasibility of using the proposed system i n university settings, it is worth noting that the set - up procedure can be completed in as little as 15 minutes by one user. All that is needed are two mobile devices (further referred to as set - up devices) activ ely generating continuous WiFi traffic (e. g. Youtube video) in add ition to one WiFi monitor. These set - address of each device should be accessible. By providing the MAC address for each device, the system monitors the surr ou nding environment only capturing packets from the specified MAC addresses, ultimately reducing the processing needed to determine the cutoff values. The idea is to use RSSI readings from set - up devices to determi ne RCVs (shown in Figure 4 - 2) which will ena bl e monitored traffic to be classified as inside or outside the room. RCV #2 for edge placements ( Figure 4 - 2b) allows traffic classified as inside the classroom to be furt her labeled as: front or back and left or rig ht . A python set - up script that automatica ll y determines the RCVs was created to aid the easy deployment. The script contains concise instructions and little human involvement is needed . A sample output of set - up script is shown in Figure 4 - 3. Figure 4 - 3 : Sa mple Output of SEMINAR Set - Up Script Ident if ying the RSSI Cu toff Value. The goal of this procedure is to obtain an RCV that will maximize the capture of traffic generated inside the classroom while minimizing the capture of traffic that is not. Two 34 key fac tors contribute to the accomplishment this g oal. For this ex planation we focus on the identification of the center monitor RCV, depicted in Figure 4 - 4a. The first key factor is the strategic corner placement of set - up devices when determining the RCV. Th e di agonally opposite corners correspond to th e furthest distanc e from the center monitor, and since attenuation increases with distance, the average signal strength of packets from a set - up device is representative of the minimum received signal strength that should be observed from packets generated i nside the classr oom. This factor maximizes the capture of traffic generated inside the class. Captured traffic from outside of the room is minimized due to the natural characteristics of wireless links where atte nuation of signal strength increases as it p ropagates throug h matter (e.g. concrete wall). The distance between the monitor and the outside of the classroom in addition to the concrete walls of the classroom effectively decreases signal strength of pac kets not generated inside the class. As a resu lt , majority of RS SI readings of outside devices are not within the acceptable range of RSSI values dictated by the RCV. The distribution of RSSI readings acquired from each set - up device is examined to determi ne a RCV such that at least 50% of packets gen er ated by each ind ividual device can be seen by the monitor is chosen. Similarly, set - up devices are placed as shown in Figure 4 - 4b for determining edge monitor RCVs; and as discussed in the previous section, two RSS I cutoff values are determined for these m on itor placements. Figure 4 - 4 : Monitor Placements for Set - Up Procedure (a) Front Monitor RCV Determination (b) Edge Monitor RCVs Determ ination 35 4.4. Dataset Analysis In this section we analyze data from the SEMINAR dataset, illustrating natural trends of WiFi traffic generated by st es . In addition, the ability of the proposed system to capture both short - term (e.g. minute to mi nute) and long - term (e.g. month to month) temporal patterns in classroom WiFi usage is presented in this section. As discussed in Section 3a, the current datas et consists of mu ltiple monitoring sessions of a single 50 - minute course with one instructor. All class notes were generated during the lecture and no online class material was provided. As a result, students are e xpected to pay close attention to lecture in order to copy the generated notes as there was no expressed need to access the Internet. There fore, our analysis highlights the ability for SEMINAR to provide significant insight into mobile device usage during lectures, ultimately leading to developmen t of a collective classroom engagement measure based on mobile device usage. 4.4.1. Validity of Collecte d Dataset Figure 4 - 5 illustrates the validity of the collected dataset, proving that the data collected is representat ive of the WiFi traffic dynamics inside of t he classroom. D uring this particular data collection, we begin collecting data at 3:25pm, 45 mi nutes before the start of the lecture that will be monitored. At the start of the collection there were already 15 st udents in the classroom waiting for the le ct ure to begin. F rom that point on we recorded the number of students that entered the room per m inute. This can be seen by the red line in Figure 4 - 5. We overlay the 4 - minute moving average of the total amount of da ta frames that remains after the filtering p rocess describe d in Section 3C. Since the lecture had not yet began, many students were active on their mobile devices. A strong correlation is observed between the number of students and WiFi activity in the cla ssroom; the volume of traffic generated pe r minute increase s as the occupancy level increases. At 4:10pm the lecture begins, as well as a d ramatic decline in the amount of traffic generated in the classroom. At 4:45pm the students in the classroom are expl icitly asked to browse the Internet on the ir mobile devices . This browsing session can also be observed in Figure 4 - 5. The class is dismissed at 5:00pm 36 and students began to exit. Due to large groups of students departing all at once, recording the number of students that departed per minute was inf ea sible. We inste ad record the time at which the departure of students concludes. At that time, t he total amount of data frames captured falls to zero. Ultimately, Figure 4 - 5 illustrates the responsiveness of the pro posed system as well as the validity of th e data collected. Figure 4 - 5 : Responsiveness of Monitor ing System 4.4.2. Weekly Patterns of Mobile Dev ice Usage The monitored course was held three times a week (i.e. Monday, Wednesday, Friday), and data collection occ urred multiple weeks throughout the semest er. Monitoring se ssions began two minutes prior to the start of class and concluded two minutes a fter the end of class. In Figure 4 - 6 traffic patterns can be observed for two different weeks in the semester, Week 10 ( Figure 4 - 6a ) and Week 12 ( Figure 4 - 6a ). As s hown in both Fig 4 - 6a and 4 - 6b, there is a time interval at the beginning of the lecture where the intensity of WiFi traffic is high and decreases throughout the first few minutes of the class. We refer to this as the Lecture Settle - in Time (LST). The LST vari es from day t o day, for Figure 4 - 6a) remained near 1600 packets per minute until about minute 9 where a sharp decline is observed. Conversely, on Wednesday of the s ame week the decline occurred much earlier tha n it did on M onday; effe ctively demonstrating that after the lecture begins, mobile device usage declines at various rates. Identifying factors that affect LST is part of our future work motivation, linking our traffic measurements to levels of engagem ent and analyzing various cl ass dynamics that may induce unengaged behavior such 37 as browsing on mobile devices during class time. As a result, we could potentially conclude that larger Lecture Settle - in Times corre ruct or beginning the lecture , instructor unpreparedness or tardiness, or simply a lack of interest or engagement for the current lecture topic. Figure 4 - 6 : Mobile Device Usage for Different Weeks In Figure 4 - 6c, t he number of students who attended lecture each of the days shown in Figure 4 - 6a and 4 - 6b is plotted. On Fridays, less student s attend the lecture. This trend is observed throughout the entire semester. Consequently, the total generated data on Fridays is si gnificantly less than it other days of the week as shown in both Week 10 and Week 12 in the figure below. In addition, the i ntensity of traffic at the start of data collection (two minutes before the lecture begins) on Fridays is significantly lower than M ondays and Wednesdays. 4.4.3. Best - Case Scenario for M aximum Stud ent Engagement Figure 4 - 7 displays four sets of data collected: Two normal Friday lectures and two exam day lectures that were given on a Friday. Several key conclusions can be derived from the res ults displayed in the figure: 1) The volum e of t raffic gene rated during normal Friday lectures is substantially more than the traffic volume generated during Friday ex ams. It should be noted that nearly all students are present on the day of exams while ne arly a fourth of students are typically no t pres ent on a re gular Friday ( Figure 4 - 6c). Even with more students present during exam day, the substantial decrease in traff ic volume on those days is expected because all mobile devices are put away during the du ration of the exam. Since mobile device us age is prohibited during exams and extremely low traffic volume on those days are observed ( Figure 4 - (a) Week 10 Traffic Patterns ( b ) Week 12 Traffic Patterns ( c ) Number of Students Present During Lecture 38 7) as opposed to normal Friday lectures where there is no restriction on mobile device usage, the validity and resp onsiveness of the proposed system is also demons trated here . 2) Exam day data provides the best - case scenario for engagement measurement. As mentioned above, mobile device usage during exams is prohibited thus Figure 4 - 7 exam days can represent a day in whi ch all students are fully engaged and ther e is n o Internet browsing. Notice that the traffic intensity does not reach zero, this is expected as many devices have background applications running. In addition, the duration of the LST on exam days is shorter when compared to normal Fridays or even ot her da ys of the w eek from Figure 4 - 6. 3) There is significant variation in traffic patterns when comp aring the two normal Fridays lectures. More specifically, during the time interval t=20 minutes to t=35 minutes, th e traffic intensity of Normal Friday #1 is notic eably great er than that of Normal Friday #2. The reason for the variation is not the focus of this work, however, it supports our hypothesis of a correlation between mobile device usage and classroom engageme nt. Ultimately, it motivates our future wo rk of using the v olume of WiFi traffic generated within a classroom as an estimator of student enga gement. Figure 4 - 7 : Comparison of Traffic Patterns on Exam Days and Normal Fridays 4.4.4. Spatial Analysis of Gener ated WiFi Traffic The proposed system has the cap ability of separating collected traffic into spatial classroom divisions (i.e. left and right). This capability is demonstrated during one of the lectures where a short eight - minute quiz was distributed acc ording to a specifi c procedure. The quiz p rocedure was as follows: 39 The quiz was distributed to the right half of the classroom, simultaneously, the left half of the classroom was instructed to browse on their mobile devices. After 8 minutes, the quiz was collected from t he right half followed by the dist ribution of the quiz to the left half of the classroom. During this time, the right side is instructed to browse on their mobile devices. After another 8 minutes the quiz is collected and the lecture resume s. Using the R SSI Cutoff Values (RCV) det ermined during th e SEMINAR set - up procedure, as discussed in Section 4 .3 , we divide the traffic into left side and right side based on the RSSI value observed from the left edge monitor ( Figure 4 - 3b). Figure 4 - 8 displa ys the traffic pattern for both sides. Tim e interv als in wh ich the different actions take place are shaded in Figure 4 - 8 and are labeled as: Right - Side Quiz Distribution , Right - Side Quiz , Right - Side Quiz Collection and Left - Side Quiz Distribution , Left - Side Quiz , and Lef t - Side Quiz Collection . It ca n be obs erved tha t during the Right - Side quiz the traffic intensity from the right side is much lower than that of the left side. Similarly, during the Left - Side quiz the traffic intensity of the right side is much higher than th e left side. The proposed s ystem ef fectively separates traffic generated by different sides of the classroom. Although there are noticeable differences in traffic intensities during each quiz interval, similarities can be observed during bot h the beginning and end of the lecture. Figure 4 - 9 d isplays t he amount of traffic generated by each side during both quiz intervals. While the total volume of traffic generated by each side throughout the entire lecture is nearly equivalent, during respectiv e quiz interval s, each side displays a dra matic de crease in traffic generation. 40 Figure 4 - 8 : Spatial Traffic Analysis During Explicit Browsing Sessions Figure 4 - 9 : Proportion of WiFi Traffic Generated Durin g Quiz Intervals 4.5. Future Work Currently, we are extending the SEMINAR dataset by coll ecting data fr om a variety of courses. The courses being monitored have diverse class attributes (e.g. course level, given course material versus no given course material, etc.) which will be used to identify simil ariti es and differences in mobile device u sage among a ll class types. SEMINAR can be effectively deployed in multiple classrooms, ultimately providing a large overview of mobile device usage during class lectures across the university. In addition, a colla borat ion with scholars specializing in ped agogy is ong oi ng to provide ground truth student engagement measures. This will ultimately ensure that the proposed system accurately reports the best estimate of student engagement levels in real - time. Lastly, imp rovem ents to the front - end of the system a re currently t aking place to provide the best user experience for instructors and university administrators. This includes a clean display of real - time data as well as easy access to previous data. 41 4.6. Conclusion Thi s worked investigated the usage of mobile devices ins i de a classroom during a class lecture using the proposed SEMIANR system. Presented in this work is ability to use the proposed system to capture different device usage patterns across di fferent days, in cluding exam days with no device usage whi ch show a b e st - case scenario of maximum engagement from the perspective of WiFi traffic measurements. Also presented are explicit browsing sessions that demonstrate the responsiveness of the system and. The system can be used to monitor device usage inside classrooms with the goal of using the data to determine a collective assessment of the engagement of students during lectures. We believe that this works opens doors to high frequency engagement me asurements, ulti mately providing an extensive student enga gement data s et that can be used to provide quality of education or quality of teaching assessment. In the next chapter, we investigate a CSI - based activity recognition system that extends contactles s exercise activ ity monitoring. 42 5 Human Activity Analysis b y Exploiting WiFi Channel State Information In this chapter , we present a system that exploits the ubiquitous WiFi signals and the correlations between signal changes and body movements to achieve contactless huma n exercise activity recognition. There is a growing tr end for people to perform regular workouts in home/office environments because of the widespread understanding of physical and mental health benefits regular physical fitness brings. Additionally, there is an increasing trend to monit or these w orkouts in a contactless fashion, free from intrusive and cumbersome wearable devices. To aid contactless monitoring of exercises, this chapter presents a non - invasive system that recognizes exercise activity and p rovides fine - grained repetition counting i nformation o f each exercise set using WiFi channel state information . Different from prior works, we attempt to accomplish accurate exercise recognition wh ile giving a higher priority to easy practical system deplo yment than typical done . In particular, o u r system aim s to detect the desired exercise with a Tx - Rx device placement representative of real - world deployment, c ontrary to standard configurations in relevant literature. Since a single workout session typical ly co nsists of multiple sets of repetitio n s, it is imp erative to be able to detect other non - exercise activities as well. Extensive data collection is performed to sufficiently train and validate machine learning classifiers. We present a novel approach to effe ctively extract exercise information embedded in received packets during the exercise and obtain both detection accuracy of pushup exercise and repetition count accuracy above 90%. Experiments show that exercise recognition can successfully be achieve d by exploiting WiFi CSI with a more flex i ble device c onfiguration that current relevant works. 5.1 Introduction E xtensive health studies [ 61 - 6 3 ] have shown that excessive sedentary behavior and physical inactivity are major risk factors for obesity, diabetes , and several cardiovascular disease s. For tunately, w orks such as [6 4 ] and [65 ] have investigated s trategies that are most effective to motivate behavior change among sedentary adults and indicate that physical activity monitors motivate physical activity and decrease sedentary behaviors . T h erefor e, reductio n in such risk factors can be achieved by incorporating physical 43 wearable devices such as Apple Watch [ 66 ] and Fitbit [ 67 ] have been developed to pr omote healthy lifestyles. Other exer cise a ctivity m onitoring solutions , such as [ 68 ], incorporate accelerometer s and other motions sensor s that are attached to the body in the form of wearable devices . A RFID - based solution is proposed in [ 69 ] to accomplis h exercise recognition. While effect ive, a ll previo usly mentioned work requires the use of a wearable device to monitor user activity and therefore are inherently limited with respect to the total amount of activity monitored as the users may forget to att ach device. Users may also find the use of a wearab le devices while exercising uncomfortable and would rather opt out. A more desirable approach would be contactless recognition of wo rkout activities. Ubiquitous WiFi infrastructures in home/office environm ents makes WiFi a reasonable sensing measu rement to accomplish the goal of exercise recognition in a contactless fashion. In fact, WiFi channel state information has proved to be an e ffective sensing modality to accomplish contactless exercise activity mon itoring and recognition of many body weight exercise s as demonstrated in a variety of recent work as discussed in Related Work, Chapter 2. While current CSI - based exercise activity recognition systems successfully achieve recognition of their vario us free weight and bodyweight exerci ses we find that curren t literature has a standard convention of Tx - Rx placement and orientation . Placement and or ientation consist of a straight - line communication link between Tx and Rx 3m 5m apa rt, and the human subject conducts e xercis e in the link. Al though contactless, this setup is quite restrictive and can become a limiting factor in practical implementation in office/home environments since creating the Tx - Rx link requires sufficient space for both devices. In actual deployme nt of CSI - based exercis e recognition systems, a desired setup is depicted in Figure 5 - 9 . Home/Office WiFi Access Points (AP), typically i n a fixed location on top of a desk or shelf, can serve as the Tx device . The Rx devi ce can be placed in a designated wor kout a rea that the orie ntation of the Rx device is inconsequential to the exercise detection. Note the referenced setup is an unconvent ional configuration with respect to current literature as the Tx is placed in practic al location resulting in nonequal di stance s from the ground . In addition, this configuration consists of device orientations where the antennas 44 of the Tx and Rx devices ar e not directly facing each other. This setup is more representative of practical impl ementation of home/office contactles s exer cise monitoring s ystems. Therefore, w e aim to extend the current literature convention of restricted device placement and orient ation when developing a system to detect or differentiate between various human exerc ises . In this work, we take contactl ess ex ercise recognitio n systems one step further by demonstrat ing that exercise data , namely the push - up exercise, can be accurately d etected and analyzed to provide exercise statistics such as the number of pushup repe titions with an more practical devic e plac ement and orienta tion , different from current literature convention. 5.2 System Design The proposed system entails several essential system functions required to achieve successful exercise recognition. Channel State Information measurements are col lected in the form of a ma trix that provides both phase and amplitude information of the incom ing signal on each receive antenna. As discussed in Chapter 1, Orthogonal Frequency Division Multiplexing modulation scheme sprea ds the data across multiple frequ encies cr eating multiple i ndependent narrowband signals to be analyzed known as subcarriers. E ach CSI stream contains readings from 56 subcarriers. Consequently, the CSI matrix contains data from all subcarrier frequencies , collected by each receive anten na , sent by each transmit antenna . In this work, we discard phase information as we are able t o extract the necessary exercise information from the CSI amplitude information alone. As illustrated in Figure 5 - 1 , the raw CSI d ata from each collection set is p rocessed through the syste m which contains two core system components. The first core componen t , the Pushup Data Classification (PDC) module, is tasked with identifying if the received signal data contains embedded pushup e xercise information by the extrac ting 6 ha nd - crafted time - s eries features. Data which is successfully classified as Counter (PDRC), tasked with detecting t he number if pushups performed du ring the respective data c ollection. 45 Figure 5 - 1 : System Overview 5.2.1 P ushup Data Classification Preprocessing and Normalizatio n : The raw CSI amplitude measurements can be affected by external factors such as hardware imperfec tion of the commodity WiFi device , interfe rence from nearby devices , as well as signal propagation properties of the ambient environment . Internal state transition s such as transmission rate adaptation and transmission power changes introduce burst noise i n the CSI streams [ 70]. For an effe ctive f eature extraction , we preprocess the CSI amplitude data t o remove any unwanted data outliers, noise, and trends. The proposed system analyzes the amplitude variance among subcarriers and antennas; therefore, we ens ure that the calculated variance is not af fected by any unw anted outlier amplitude values by applyi ng a Hampel filter to remove outlier values. Specifically, we apply the Hampel filter with a sliding window at each subcarrier to remove the outliers wh ich have significantly different am plitude values from other nei ghboring subcarriers. This filter e ffectively eliminates outliers and reduces signal noise. The CSI signals sometimes display a trend, which can be visualized as a positive or negative slope o ver the length of the signal . For thi s reason, we detrend the ra w amplitude data by subtractin g the mean amplitude value from each received packet in the current data collection set. Unity - based normalization is performed where all values are scaled i nto the range [0 1]. This step make s the s ystem agnostic to specific 46 subcarrier a mplitude values ob served from day to day or location to location and instead focus on the amplitude variation between subcarriers. Data Classification : The data classification breakdown can be observe d in Figure 5 - 2 . Th e only desired data is the pushup data collection set. As shown, the collected data can be grouped into one of two high - level classes Activities on Mat and No Activity on Mat. Data can be further classified into fo ur different classes. Empty Room da ta co n s ists of CSI data collected when the data collection area is free from human subjects. Lay Still data consist of CSI data collected when human subjects laid on the workout mat with little to no movement. Various mov ements conducted on the workout mat such a s, but not limited to, stre tching, browsing on phone, and enter ing/ depart ing collection area, are classified as Misc. Movements data. The final class of data is Pushup data where the data collection set contains em bedded exercise information. While the p us hup data is the only data o f interest to be recognized, multiple data classes are needed to efficiently separate pushup data from other types of activity. We show in Figure 5 - 12 the benefit of constructing a binary c lassifier by combining outputs o f t he mult i - class classifier. We appl y the k - Nearest Neighbors algorithm (kNN) to classify the processed data. The multi - class classifier output is then converted to a binary output where one class is the Pus hup Activit y and the other is No Pushup Activi ty. Any data not classified as pushup a ctivity is grouped into the No Pushup Activity class. Pushup activity data is then further analyzed by the PDRC. Figure 5 - 2 : Classification Breakdown 47 Figure 5 - 3 displays the unique heatmap signatures of collected CSI data f rom different activities. The vi sual difference in the heatmap signatures help motivate our choice of extraction features. Figure 5 - 3 : Heatmaps of Different Activities Feature Extraction : We extract six hand - cra fted features from each data collec tion se t to construct a feature set for the classifier. Three features aim to extract the variation behavior between subcarriers , n amely, Mean Subcarrier Variance, Mean Subcarrier Standard Deviation, Mean Subcarrier Movin g Variance of each antenna. Three a ddition al features aim to extract simil arity measures between the three receiver antennas using Dynamic Time Warping (DTW). DTW is an algorithm used to measure the similarity between two data series . We measure the similarity of the signal received b etween each combination of two receive anten nas. Since there are three possible combinations of the three receive antennas, we obta in three similarity measures to use as addition features. Figure 5 - 4 provides a visualizatio n of feature separation due to the specifi ed features. Figure 5 - 4 a, 3 of the 6 fe atures are used to illustrate the feature separation between the 4 classes. Specificall y, the mean MSSD and MSMV across antennas, in addition to the DTW A2,A3 similarity measure , were used. Fig 5 - 4 b provides an ad ditiona l perspective of the feature separati on as 48 another 3 combination of features are used to illustrate separation in the differ ent classes of data. Opposite from Figure 5 - 4 a, this feature separation figure consists of t wo similarity measure features and one var iation related experience. Classifica tion results are presented in Section 5.3.1 . Figure 5 - 4 : Feature Separation Virtualizations 5.2.2 Pushup Data Repetition Counter The purpose of this module to extract workout statis tics from the processed data collect ion se t. Specifically, this module detects the number of pushups performed by the human subject by ex tracting out the periodic signal, created by human motion, that is embedded in the received signal. Packet Correlatio n Matrix (PCM) : The first step to ex tracti ng the repetition information out of the classified pushup data is to construct a packet correl ation matrix. For each packet in the set, a correlation coefficient is determined for every other packet in the data co llect set, effectively creating the correl ation matrix. The idea behind this st ep is to exploit the stability and consistency of CSI amplitude values for a given location. Consider the pushup exercise sequence as shown in Figure 5 - 5 . Given a receiver at a fi xed nearby location collecting CSI d ata as discussed in Data Collection section , as a human subject completes t he exercise sequence in Figure 5 - 5 (a) - (e), we expect that majority of subcarriers will display similar amplitude values while in UP posi tion (e.g. Figure 5 - 5 a,c ,e ) and similar amplitude value s while in DOWN position (e.g. Figure 5 - 5 b,d), respectively. 49 Figure 5 - 5 : Pushup Exercise Sequence Figure 5 - 6 provides packet correlation matrices for 4 different pushups collect sets. Note that the data collection begi ns in the U P position as in Figure 5 - 5 a. It can be observed that the PCM is able to clearly illustrate a periodic motion across time. Depicted i n Figure 5 - 6 a, specifically for packet indices 0 to 1000, is a group of subsequent received packets that are highly correlated with respect to subcarrie r ampl itude values for the initial 480 pack ets, approximately, indicated by the darker colors in the figure. To map that data to physical human movement, the initial 400 packets are highly correlate since the human subje ct is in th e UP position, effectivel y crea ting a distinct consistent multipath signal. After roughly 400 packets, the next batch of received packets (e.g. approximately packet indices 401 through 550) began to correlate less with the initial batch of recei ved packets. This data maps to when the hu man subject moves from the UP to DOWN position. A noteworthy observation from Figure 5 - 6 is the diversity in pushups performed. While Figure 5 - 6 a shows six pushups performed at a near fixed rate , Figure 5 - 6 c shows a push up e xercise session where the subjec t init ially began fast and completed the la st half of the set at a slower pace. 50 Figure 5 - 6 : Pushup Data Packet Correlation Matrices After the PCM is calculated, the mean signal is obtained by averaging all correlation values in each column of the matrix . The result, as displayed in Figure 5 - 7 a , is the average packet correlation signal which encompasses the desired exercise information. This data is a continuation of Packet Correlation Matrix #1 in Figure 5 - 6 a. F urther proc essing is required to accurately det ect th e number of pushups performed . We imp lement a cascade of filters to clean the signal in preparation for the peak detector. First, a lowpass filter is utilized to discard unwanted high frequency noise and the result ing signal is displayed in Figure 5 - 7 b . Next , a median filter is used to smooth t he signal in order to obtain a rough outline of the period motion as the packet index increases from zero. This can be visualized in Figure 5 - 7 c . The data is s moothed using Savitz ky Golay filter which computes the l ocal p olynomial least square fitting in the time domain to filter out noise while ensuring that the shape and width of the signal are unchanged . Thus, the Savitzky Golay enables CSI signal denoi s ing without distort ion of the signal waveform. After a final moving average is implemented to eliminate any unwanted peaks in the signal, a peak detection algorithm is performed . As depicted in Figure 5 - 7d, the final signal is a periodic signal that represe nts the periodic hum an motion 51 during the pushup exercise . A gr een box located on each peak illustrates su ccessfully detected peaked in the signal. At this stage, the number of detected peaks will serve as the number of pushups performed. An additional enti re detection process is provided in Figure 5 - 8. Figure 5 - 7 : Push up Detection Counter Data Processing 52 Figure 5 - 8 : Full Pushup Detection Counter Data Processing 5.3 Evaluation In this section, we present the implementation and evaluation results of the proposed exercise recognit ion system. Hardware Setup. The proposed s ystem incorporates two PCs running Ubuntu 1 4.04 LTS (64bit) with Linux kernel is 4.1.10. One device serves as the AP and Tx device and the second device serves as the Rx, as illustrated in Figure 5 - 1. Both devices ha ve an Atheros AR9580 Network Int erface Car d ( NIC ) installed, modified using Atheros C SI Tool . Tx is eq uipped with two transmit antennas with a p acket transmission rate of 200 packets per second. R x is equipped with three receive antennas and continuou sly captures the 53 incoming packet s for late r processing. As a result, we exploit the variat ion in CSI subcarrier values across the multiple receive antennas to extract pushup information.]. Environment and Data Collection. We consider a scenario where the human activities are monitored i n multiple indoor enviro nments using Tx and Rx devices dis cussed in previous section. Figure 5 - 9 illustrates an indoor home environment where Rx is placed on the ground in a designated exercise area while Tx, the AP, is locate d in designated work area on top of a desk . The unique a spect of our collection setup is t he how Tx and Rx are separated in space. A standard Tx - Rx placement, as seen in many exercise recognition works, consists of both devices in the middle of the room on the floor. The subject will the n conduct the exercise b etween the two devices. While effe ctive, that standard placement can be a limiting factor in deployment of practical exercise monitoring systems. Consider an XYZ coordinate system, while current liter ature separates the Tx and Rx de vices only on a single a xis (e.g. x - axis, y - axis), the pro posed system, however, is designed to be practical and flexible where the Tx - Rx devices may be placed with separation on all three axis. Figure 5 - 9 illustrates the sepa ration in the x - axis (e.g. 4 met ers), sepa ration in the y - axis (e.g. 3.5 meters), and sepa ration in the z - axis (e.g. 1 meter). We demonstrate that this unconventional data collection setup is does not impede our ability to obtain periodic exercise informat ion. One data collection consis ts of a 15 - 30 second ses sion in which 1 of 4 human activit ies, discussed previously is being performed. The collected data is obtained from 7 participants, each conducting multiple data collections effectively creating 500 t otal collection sets of activity data. Pus hup collection sets consisted of pushups ranging from 5 to 12 repetitions per set. When performing pushups, the subject begins position at w hich point the data coll ection ends. 54 Figure 5 - 9 : Data Col lection Environment and Device Placements 5.3.1 Pushup Data Classification In this section, we evaluate the pushup data classification performance . To evaluate the perform ance of our classification system we use t hree evaluation metrics , k - fold cross validation , c onfusion m atrix , and a ccuracy. k - fold cross validation splits the data set into k smaller sets and a model is trained using the folds and validated on the remaining part of the data . In our impleme ntat ion k = 10 . We ensure robust training by mix ing and shuffl ing the collections from all participants on different days. It is important to recognize that due to subtle changes in the static environment, data fro m the same participant may exhibit less co nsistency. Therefore, we use the diverse data co llected to train a kNN model with Euclidean distance metric and number of n eighbors equal to 6. After 10,000 runs, the average cross validation loss reported was .047 3 . Figure 5 - 10 display s mean confusion matri ces calculated after 10,000 runs. Initially, a m ulti - class classification is performed. This step benefits our overall obje ctive of separating pushup activity data from all other classes of activity data when on th e workout mat. The full set of non - pushup activities have both similar and distinct featur es. It proved advantageous to group the different kinds of non - pushup activ ities by exploiting their distinct features, for an initial layer of classification. Figure 5 - 10a displays the confusion matrix of the multi - class classification performance. All 4 cl asses can be distinguished with an accuracy greater than 85%, three of whic h above 90%. Most importantly, the system detects approximately 95% of all pushup data. Alt hough the non - pushup activi ties are not of interested in the current work, the 55 ability to classify the other activities while on the workout mat aids the long - term s ystem design objective discussed in Future Works section. Figure 5 - 1 0 : Confusion Matrix fo r Pushup Data Classifier Pe rformance Since there exists exactly one data class of interest , the task of filtering non - pushup activity data collection sets is effectively a binary classification problem. Figure 5 - 10b provides the binary confusion matrix after converting multi - class res ults to binary. Similar to the positive detection performance o f pushup activity, the system classifies 95.41% if non - pushup activi ty correctly. It should be noted that while the system incorporates an additional classification s tep, the outcome is an incr eased detection accuracy. As shown in Figure 5 - 11 , when using the same six features to directly construct a binary classifier where t he two classes of data are non - pushup activity and pushup activity, the overall performance is des irable. Pushup and non - push up data are det ected with approximately 90% and 96% accuracy, r espectively. While pushup detection accuracy of pushup data is 90%, the pushup data detection accuracy via multi - class classification achieved a higher detection accu racy of nearly 95% . Not onl y is the 95% to tal accuracy of the binary classifier after mult i - class output conversion higher than that of the 93% total accuracy native binary classifier, as displayed in Figure 5 - 12; the native binary classifier is less desirab le simply because the amoun t of pushup dat a loss when moving to next stage in the exercise recognition system is greater. As a result, the proposed system imp lements a multi - class classifier and converts its outputs into a binary classification problem. 56 Figure 5 - 1 1 : Confusion Matrix for Pushup Dat a Classifier Performance without Multi - Class Out puts Figure 5 - 1 2 : Classifier Performance 5.3.2 Pushup Repetition Counter In this section, we evaluate the pu shup data classifica tion performance. We thoroughly evaluate the performance of the proposed pushup rep etition counter by considering the following eva luation metrics : Pushup Repetition Precision , where is the total number of repetitions detected by the system and is the actual total number of pushup repetitions from all collection sets. This accuracy reflects the overall ability of the system detect a pushu p repetition when given a pushup repetition and provides re liability assurance. Perfect Repetition Count Accuracy (P R C A ) . , where represents the number of detected sets , where the estimated number of repetition an d match the expected number of repetitions for a particular set. is the total number of pushup collection sets. The PRCA value should be as high as 57 possible since PRCA reflects the ability of the system to detect all pushups performed by a user d uring data collection set . Imperfect Set Repetition Error (EPRE ). EPRE is the expected set repetition count error . This error value provides the expected difference in the number of repetitions detected by and the actual number of repetitions performed pe r data collection set . This evaluation metrics looks at how much the detected repetition count is expected to diff er from the actual count per set given that the set did not report the current number of repetitions. End - to - End Repetition Pushup Repetition Precision (E P R P ) EPRP is the pushup repetition count preci sion after the data is processed end - to - end. This means that EPRP depends on the systems ability to correctly classify pushup data as pushup data and non - pushup data as non - pushup data. Table 5 - 1 p rovides the performance results obtained from experimental analysis. The pushup repetition precision is 98.24%. This demonstrates that the proposed system is able to accurately count the number of repetitions performed while using the proposed device pla ce ment framewo rk. We also calculate the PRCA. The proposed sy stem detects the correct number of pushup repetitions in 92% of the pushup collect sets it encounters and when the detected number of repetitions in not correct is off by ± 1.08 r epetitions . The af orementioned results were with respect to the pushup detect or as a stand - alone mechanism. We additionally provide end - to - end performance results where for a pushup repetition to be succes sfully detected, the data collection set must first be classified a s pushup data. The end - to - end pushup repetition precision is 96%. This is only slightly less than the expected 98% of the pushup repletion counter and means that the pushup data classifier effectively pushes through to repetition counter only pushup data, wh ile minimiza tion other activity data. Since there is a smal l amount of non - pushup activity that gets passed through, the overall repetition count error expected per set when the repetition count was not perfectly detected is ± 2 . 92 r epetitions . 58 Table . 5 - 1 : Pushup Repetition Counter Performance 5.4 Limitation It should be noted that the pushup repetition counter module performance suffers when encountered with a problem scenario. In complex environments, multiple multipath CSI signatures can be received wi t hin a very short time window. Figu re 5 - 13a illustrates this phenomenon. Some collection environments are inherently rich in multipath. When faced with this situation, the packet correlation matrix constructed as in Section 5.2.2 , does not provide an accur at e depiction of the periodic human motion during the exercise. This is a natural effect of receiving multiple signatures because two different received CSI signatures may already be unique, thus the change in correlation values in the data does not solely c ontain variation information caused by human movements. The negative effect of multiple signatures with respect to the packet correlation matrix can be viewed in Figure 5 - 13b. When compared to packet correlation matrices discussed in Section 5.2.2 the push up s in Figure 5 - 13b are less distinguishable. As discussed in Future Work, we aim to extend system capabilities, including the ability to overcome multiple signatures obtained from complex wireless environments in which the aforement ioned phenomenon is exten si vely investigated. Currently, i n this work we report repetition detection results after discarding collections with multiple signatures. 59 Figure 5 - 1 3 : Multiple CSI Signatures 5.5 Future Work and Dis cussion Our plan to extend the current work involves sever al components. System Robustness . As discussed in section 5.4, currently system is not robust in the event of sever multipath effect and multiple CSI signatures are present in the collecte d data. W e most incorporate further processing to effectively separ ate the signatures. It can be seen in Figure 5 - 13a the when multiple signatures are formed, each CSI signature is generally consistent, therefore, some clustering method can be used to distin guish wh i ch packets are associated with which signatures. This is i mportant since each signature conta ins exercise information embedded within. Additionally, we will have to ensure that while analyzing the individual signatures that the overall exercise ti me - relat e d information is preserved for later exercise statistics e xtraction . Given that our objective is practicality, we must ensure the system is r obust in different real - world scenarios and settings. For example, there could be multiple persons or othe r moving objects around. The person or other objects could block th e direct path between the transmitt er and receive r; or the person can be moving in the vicinity of the exerciser during the workout, which could affect the recognition accuracy of the syste m. Multi - Person Workout Sessions . Although the proposed system is d esigned for a single person, a real - world scenario can consist of multiple users conducting exercise together as shown in Figure 5 - 14. Since the 60 received signal at each receiver is independen t, each R x device should be able to extract nearby exercise data, u naffected by the exercise movements of another person. Initial experiments support our hypothesis that we can obtain individual workout statistics with the device placement in Figure 5 - 14. No te, that the device placement is an extension of the currently pr op osed device placement. We plan to systematically analyze and model system parameters such as the minimum distance required between two neighboring exercising subjects. Figure 5 - 14 : Mult i - Pe rson Wor k out Scenario Multi - Exercise Recognition System . Related w orks such as [7 1 - 7 2 ] show that it is possible to classify various exercises (e.g. pushups, sit - ups, and squats). This was not the objective of the proposed work, however, to confiden tly clai m and demonstrate that the proposed device placem ent and any other similar practical setup can achieve same performance as state - of - the - art cumbersome device placement , we must incorporate a multi - exercise classifier. In addition, we plan to add to the l ist of workout statistics provided. Currently, we provide only rep etitions detected. We can add abilities such as measuring t he t ime i nterval between e ach workout a ctivity or a ssessing if the e xercise s were p erformed p roperly . Altogether, consider group ex ercise classes where individuals must remain in their provided wor kout space and follow the instructions of the workout instructor. A small Rx device can be placed nearby, and the workout analysis can be provided, and even co mpared to the instructo ormance. 61 5.6 Conclusion In this work, a contactless exercise recognit ion system exploits the unique variations in the wireless channel state of individual wireless receivers caused by nearby human movements was proposed . More specifically, we prop osed a syst em that employs various signal processing techniques to extract ou t the nearby cyclic al human movement information embedded into the received wireless signal s , ultimately provid ing contextual information for specific human exercise activities. T he collect e d data cannot directly be used to obtain fine - grained activity inf ormation, so we des ign a series of data denoising and smooth ing methods prior to extracting hand - crafted features to distinguish pushup data from the other classes . Ultimately, th e proposed system can confidently perform fine - grained exercise recognition , namely the pushup exercise, and provide exercise statistics such as the number of sets and repetitions without the need of wearable device. 62 REF ERENCES 63 REF ERENCES [1] Chen, Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. "A survey of depth and inertial sensor fusion for h u man action recognition." Multi media Tools and Applications 76.3 (2017): 4405 - 4425. [2] Zhang, Yuhe, and Lin Zhang. "WiFi - based contactl ess activity recognition on smartphones . 2017 IEEE/CIC International Conference on Communications in China (ICCC) . IEEE, 2017 . [3] Chowdhury, Tahmid Z. Using Wi - Fi channel state information (CSI) for human a ctivity recognition and fall detection . Diss. University of British Columbia, 2018. [4] Cao, Yangjie, et al. "Contactless Body Movement Recognition during Sleep via WiFi Signals." IE EE Internet of Things Journal (2019). [5] Huang, Si, et al. "Wiga: A WiFi - Based Contactless Activity Sequence Recognition System Based on Deep Learning." 2019 15th International Conference on Mobile Ad - Hoc and Sensor Networks (MSN) . IEEE, 2019. [6] Zhuang, Yuan, e t al. "Smartphone - based indoor localization with bluetoo th low energy beacons." Sensors 16.5 (2016): 596. [7] B S urvey on the state of the art and the 802.15. 4 and z 30, no. 7, pp. 1655 1695, 2007. [8] D. M. Dobkin, The RF in RFID: UHF RFID in Practice, 2nd ed. Newnes, 2012. [9] Int ernational Convention on Information and Co mmunication Technology, Electronics and Microelectronics (MIPRO). IEEE, 2018 [10] Lemic, Filip, et al. "Enriched Training Database for improving the WiFi RSSI - based indoor fingerprinting performance." 2016 13th IEEE An nual Consumer Communications & Networking Conference (CCNC) . IEEE, 2016. [11] Yang, Jianfei, et al. "Fine - grained adaptive location - independent activity recognition using commodity WiFi." Wireless Communications and Networking Conference (WCNC), 2018 I EEE. IEEE , 2018. 64 [12] Zafari, Faheem, Athanasios Gkelias, and Kin Leung. "A survey of indoor localization systems and t echnologies." arXiv preprint arXiv:1709.01015 (2017). [13] Halperin, Daniel, et al. "Tool release: Gathering 802.11 n traces with chann el state informatio n." ACM SIGCOMM Computer Communication Review 41.1 (2011): 53 - 53. [14] J . Sohl - Dickstein , S. Teng, B. M. Gaub, C. C. Rodgers, C. Li, M. R. DeWeese, and N. S. Harper, Device for Human Ultrasonic Echolocation IEEE Transactions On Biomedical Engineering , vol. 62, no. 6, 2015. [15] L. Liu, M. Popescu, M. Skubic, M. Rantz, T. Yardibi, and P. Cudd ihy, on Doppler radar motion signature In Pervasive Computing Technologies for Healthcare (Pervasive Health), vol. 222, no. 225, pp. 23 - 26, 2011. [16] P. Arlotto, M. Grimaldi, R. Naeck, and J. Ginoux, s Sensor for Breathing Monitoring Sensors , no. 8, pp. 15371 - 15386, 2014. [17] B. Raj, K. Kalgaonkar, C. Harrison, and P. Dietz, In IEEE Perva sive Computing , vol. 11, no. 2., pp. 24 - 29, 2012. [18] M. Siddique, r and its applications In Proc. of the 9th WSEAS International Conference on Applied Informatics and Communications, World [19] M. Zhou, M. Ma, Y. Zhang, K. SuiA, D. Pei, and T. Mo Measurements via Large - scal Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing - , 2016, pp. 316 327. [20] [21] E. K alogianni et al. Proceedings of The 18th AGILE International Conference on Geographic Information Science; Geographics Information Science as a n Enabler of Smarter Cities and Communities, Lisboa (P ortugal), June 9 - 14, 2015; Authors version , 2015. [22] - Situ 016, pp. 1 15. 65 [23] 2015 9th International Conference on Sensing Technology (ICST) , 2015, pp. 844 849. [24] or i n 144. [25] for resistance training based on Comput., vol. 17, no. 4, pp. 771 782, Apr. 201 3. [26] H. Ding, J. Han, L. Shangguan, W. Xi, Z. Jiang, Z. Yang, Z. Zhou, for free - w eight exercise monitoring nsactions on Mobile Computing, vol. 16, no. 12, pp. 3279 3293, Dec 2017. [27] W. Zhao, H. Fen Kinect based rehabilitation exercise 765. [28] X. Li, D. Zhang, Q. Lv, J. Xiong, S. Li, Y. Zhang, and H. Mei, - free indoor human tracking w ith commodity wi - Proc. ACM Interact. Mob. Wearable Ubiquitous Technol ., vol. 1, no. 3, pp. 72:1 72:22, Sep. 2017 . [29] Hao Wang, Daqing Zhang, Yasha Wang, Junyi Ma, Yuxiang Wang, and Shengjie Li. 2017. RT - Fall: a real - time and contactless fa ll detection system with commodity wifi devices. IEEE Transactions on Mobile Computing (2017), 511 526. [30] Ju Wang, Hongbo Jiang, Jie Xiong, Kyle Jamieson, Xiaojiang Chen, Dingyi Fang, and Binbin Xie. 2016. LiFS: Low Human - Effort, Device - Free Localization wit h Fine - Grained Subcarrier Information. In Proceedings of the 22nd Annual International Conference o n Mobile Computing and Networking, MobiCom 2016 . ACM, New York City, New York, 243 256. [31] WeiWang, Alex X. Liu, and Muhammad Shahzad. 2016. Gait Recognition Us ing WiFi Signals. In Proceedings of the 2016 ACM International Joint Conference on Pervasive a nd Ubiquitous Computing, UbiComp . ACM, 363 373. [32] WeiWang, Alex X. Liu, Muhammad Shazad, Kang Ling, and Sanglu Lu. 2017. Device - free Human Activity Recognition Usin g Commercial WiFi Devices. IEEE Journal on Selected Areas in Communications 35 (2017). Is sue 5. [33] Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E - eyes: device - free location - oriented activity identification using fine - grai ned wifi signatures. In Proceedings of the 20th annual international conference on M obile computing and networking . ACM, 617 628. 66 [34] Yuxi Wang, Kaishun Wu, and Lionel M. Ni. 2017. WiFall: Device - Free Fall Detection by Wireless Networks. IEEE Transactions on M obile Computing 16 (2017), 581 594. Issue 2. [35] Wang Wei, Liu Alex X, Shahzad Muhammad, Ling Kang, and Lu Sanglu. 2015. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of the 21st annual international conference on m obile computing and networking . ACM, 65 76. [36] Dan Wu, Daqing Zhang, Che nren Xu, Hao Wang, and Xiang Li. 2017. Device - Free WiFi Human Sensing: From Pattern - Based to Model - Based Approaches. IEEE Communications Magazine 55 (2017). [37] Dan Wu, Daqing Zhang, Chenren Xu, Yasha Wang, and Hao Wang. 2016. WiDir: walking direction est imation using wireless signals. In Proceedings of the International Joint Conference on Pervasive . ACM, 351 362. [38] Fu Xiao, Jing Chen, Xiaohui Xie, Linqing Gui, Lijuan Sun, and Ruchuan Wang. 2018. SEARE: A System fo r Exercise Activity Recognition and Quality Evaluation Based on Green Sensing. In IEEE Transactions Emerging Topics in Computing . IEEE. [39] Ning Xiao, Panlong Yang, Yubo Yan, Hao Zhou, and Xiang - Yang L i. 2018. Motion - Fi: Recognizing and Counting Repetitiv e Motions with Passive Wireless Backscattering. In IEEE International Conference on Computer Communications . IEEE. [40] Tong Xin, Bin Guo, Bin Guo, Pei Wang, Jacqueline Chi Kei Lam, Victor Li, and Zhiwen Yu. 2018. FreeSense: A Robust Approach for Indoor Hu man Detection Using Wi - Fi Signals. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , Vol. 2. ACM. Issue 3. [41] Yanni Yang, Jiannong Cao, Xuefeng Liu, and Kai Xing. 2018. Mul ti - person Sleeping Respiration Monitoring wi th COTS WiFi Devices. In 15th International Conference on Mobile Ad Hoc and Sensor Systems (MASS) . IEEE, 37 45. [42] Nan Yu, Wei Wang, Alex X. Liu, and Lingtao Kong. 2018. QGesture: Quantifying Gesture Distance and Di rection with WiFi Signals. In Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) , Vol. 2. ACM. Issue 1. [43] J. K. Aggarwal , and M. S. Ryoo. "Human activity analysis: A review." ACM Com puting Surveys (CSUR) , vol. 43, no. 16, 2011. [44] C. C. Yang, and Y. L. Hsu, - based wearable motion detectors for physical activity monitoring Sensors , vol. 10, no. 8, pp. 7772 - 7788, 2010. 67 [45] M. Swan, "The quantified self: Fundamental disruption in big da ta science and biological discovery", Big Data , vol . 1, no. 2, pp. 85 - 99, 2013. [46] M. S. Patel, D. A. Asch, and K. G. Volpp. "Wearable devices as facilitators, not drivers, of health behavior change", Jama, vol. 313, no. 5, pp. 459 - 460, 2 015. [47] D. Ledger, de Wearables Industry Report by Endeavour Partn ers , 2014. Available: http://endeavourpartners.net/white - papers/inside - wear ables [48] G . Blumrosen, B. Fishman, and Y. Yovel, and Class ification IEEE Sens ors Journal , vol. 14, no. 11, 2014. [49] Arduino Single Board Computer, Available: https://en.wikipedia.org/wiki/Arduino [50] Ma xbotics High Performance Ultrasonic Sensors, A v a i l a b l e : h ttp:// www.maxbotix.com/ MB10 [51] [52] [53] J. - S. een Student Engagemen t and Academic Performance: Is It a Myth 185, May 2014. [54] Eng Psychol., vol. 50, no . 1, pp. 1 13, Jan. 2015. [55] [56] A New Tool for Measuring St udent Behavioral Enga [57] [58] calization Systems and Tech ArXiv17090 1015 Cs, Sep. 2017. 68 [59] - based wireless indoor localization, From one to crowd: a survey on crowdsourcing - Sci., p. 0. [60] A. Mirak ccupancy behavior ba sed model predictive control for building indoor climate 513, Oct. 2016. [61] Lavie, Carl J., et al. "Sedentary behavior, exercise, and cardiovascular health." Circula tion research 124.5 ( 2019): 799 - 815. [62] Lewis, Zakkoyya H., et al. "The utility of wearable fitness trackers and implications for increased engagement: An exploratory, mixed methods observational study." DIGITAL HEALTH 6 (2020): 2055207619900059. [63] Sullivan, Al ycia N., and Margie E . Lachman. "Behavior change with fitness technology in sedentary adults: a review of the evidence for increasing physical activity." Frontiers in public health 4 (2 017): 289. [64] Cairns, Ashley, and Anne Mounsey. "Do fitness trackers use b y adults increase phy sical activity?." Evidence - Based Practice 21.6 (2018): 6. [65] Brinton, Julia E., et al. "Establishing Linkages Between Distributed Survey Responses and Consumer Wearable Device Datasets: A Pilot Protocol." JMIR research protocols 6.4 (2017 ): e66. [66] I. Pernek, K. for resistance training based on Personal Ubiquitous Comput. , vol. 17, no. 4, pp. 771 782, Apr. 2013. [67] (2017). [ Online]. Available: https://www.fitbit.com/ [68] F. Xiao, Z. Wang, N. Ye, R. Wa More Tag Enables Fine - Grained RFID Localization and sactions on Networking, DOI:10.1109/TNET.2017.2766526. 2017:1 - 14. [69] Xiao, Fu, et al. "SEARE: A system for exercise activity recognition and quali ty evaluation based o n green sensing." IEEE Transa ctions on Emerging Topics in Computing (2018). [70] Zhang, Fusang, et al. "Towards a diffraction - based sensing approach on human activity recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technolog ies 3.1 (2019): 1 - 25. 69 [71] Li, Shengjie, et al. "WiFit: Ubiquitous bodyweight exercise monitoring with commodity Wi - Fi devices." 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & C ommunications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (Smar tWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) . IEEE, 20 18.