Wearables for health and fitness tracking continue to gain popularity: an estimated 84.5 million devices were reported being used in 2019 [1] and this number doubles each year [2]. With growing interest to track workouts and sport sessions, companies such as Amazon, Apple, Google, Samsung, and Fitbit are developing various hardware sensors and software suites to provide users a holistic experience to track and document their workouts. While effort has been made to track and improve workout sessions, the analytics are based on gross assessments pertaining to basic activities, e.g., step counts, or physiological signals, such as pulse rate.

This research explores the design of a wearable platform that revolves around the use of our hands in performing activities. By understanding what the hands interact with, we can recognize, analyze, and recall the multitude of interactions we have with our surrounding environment. Previous work in computer vision, machine learning, and signal processing for predicting and detecting human activities has limited camera placements to the environment or on the body [3-7].

We propose a wrist-mounted camera pointed toward the hand and fingers to capture the user’s nuanced interactions with the environment. This will facilitate the application of advanced machine learning and computer vision techniques for activity understanding, performance assessment, as well as enable rich feedback for users including replays and performance reports. For this seed grant, we aim to investigate the use of this technology to augment training for boxing. We plan to use the above-mentioned technology to detect gestures such as the jab, hook, block, uppercut, overhead punch, and guard stance, and provide metrics like count, speed, power, pace, and accuracy. We hypothesize that the metrics along with video playback will help athletes improve their routines.

This project will provide a platform for individuals to track, monitor, and improve their sport and fitness sessions. The platform can be extended to throw and catch sports such as discus throw and Javelin throw; target sports like axe throwing and bowling; long session sports like biking, rowing, and skiing; bat and ball sports like baseball and cricket where the pitching, bowling, and swing actions can be monitored. There is also much potential for this device in the context of weightlifting and gym workouts for tracking posture and hold.

This technology can also be used by coaches to monitor and record events that need to be reviewed. Through the proposed case study, our intent is to demonstrate the feasibility of a proof of concept, thereby paving the way for a plethora of fitness and sports applications. The requested seed grant money will enable us to complete the proposed research, gather results that demonstrate feasibility, and apply for external federal grants, including NSF Smart and Connected Health, to expand this research project.

The proposed technology will provide a platform for researchers to explore questions across all strategic areas, from social impacts to participation across demographics. We are particularly interested in exploring how this technology will impact sport and fitness across personal training.

Last updated April 2021.