Automatic Recognition of Diet, Physical Activity and Sedentary Behavior Using a Smart Wearable Device

University Of Pittsburgh At Pittsburgh, Pittsburgh PA

Investigators

Linked publications, trials & patents

Abstract

Automatic Recognition of Diet, Physical Activity and Sedentary Behavior Using a Smart Wearable Device Lifestyle and behavioral factors, such as diet, physical activity, and sedentary behavior, play important roles in human health. A recent report in Lancet has found that, globally, among the top-17 risk factors of human diseases, ?diet risks?, ?child and maternal malnutrition? and ?low physical activity? ranked No.1, No. 3 and No. 14, respectively. Due to the increased adoption of unhealthy lifestyle, such as high-calorie, low-nutrient diet and decreased physical activity, chronic diseases are rising rapidly. For example, from 1980 to 2014, worldwide diabetic cases increased from 108 million to 422 million, and Americans? life expectancy declined in 2015 for the first time since 1993. In order to assist the research community to study people?s lifestyle and develop effective interventional strategies, especially for chronic diseases such as diabetes to control their diet and increase physical activity, we have developed the eButton. The eButton is a miniature wearable computer pinned on the chest region to assess diet, physical activity and sedentary behavior. After several generations of redesigns, the eButton has now been shrunk to the size of an Oreo cookie while its functionality has greatly increased. We have conducted several studies evaluating the eButton with both adults and children. Our studies have shown that this device is well accepted by human subjects, and is suitable for collecting passively and conveniently large amounts of lifestyle data in real-life. However, post processing of field-collected data has been time-consuming, mainly due to required manual procedures to review thousands of images for identification of eating and other events of research interest. Privacy concerns were also expressed by some participants on recording images in their living and working environments and manual review of the recorded images. These significant problems must be solved before this important tool is utilized for large-scale clinical studies. Powerful solution to solve these problems can be found in the most recent advances in artificial intelligence (AI). By equipping the eButton with the AI capacity, we can effectively transform this passive data collection device to a mini-robot that travels with the human subject, observes the world around it, takes quick notes (in English text) at an astonishing speed, stores these notes in a compact form within the eButton, and discards raw images immediately except for food images which may need a dietitian to analyze. If this new concept can be successfully implemented, there will be no need to review thousands of images, and the risk of privacy breach is avoided because sensitive images are no longer stored within the device. Since the eButton will only be taught to recognize a limited number of dietary activities, physical activity and sedentary behaviors, this device will produce the desired information of research interest, but will never cross the privacy intrusion boundary. In this application, our team of experienced engineers and lifestyle scientists proposes to implement this new concept. We will develop an efficient human activity recognition algorithm personalized for each human subject. This algorithm utilizes both the subject?s daily routine information and the information extracted from selected sensors. The data from these sensors will be examined sequentially to test the validity of a number of candidate activities. We will develop miniature electronic circuits to implement a deep convolutional neural network (CNN) within the eButton. This field- programmable gate array (FPGA) based implementation will enable real-time robotic reading of images and integrate the textual notes produced by the CNN with the personalized human activity recognition algorithm. Finally, we will perform a thorough evaluation study using human subjects to determine the accuracy of our activity recognition approach.