The CBA lab will attend and participate in this year’s Ubicomp / ISWC conference, which will happen from October 5 through 9 (Melbourne, Australia). Come find us at the conference and let’s have a chat on all things Ubicomp and how we can work together.

Overview

  • 4 papers
  • 1 tutorial
  • 1 keynote (TP at Wellcomp)
  • PACM IMWUT editorial board (TP)
  • Steering committee ISWC (HH, TP)

Papers

Leng et al., Ubicomp 2024 (PACM IMWUT)

  • Paper Title: IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
  • Authors: Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, Thomas Plötz
  • Type of paper: PACM IMWUT
  • Summary: One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well – as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.

Hwang & Leng et al., ISWC 2024

  • Paper Title: More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode Detection
  • Authors: Sungjin Hwang (Co-first Author), Zikang Leng(Co-first Author), Seungwoo Oh, Kwanguk Kim, Thomas Plötz
  • Type of paper: ISWC Note
  • Summary: Transportation mode detection (TMD) for wheelchair users is essential for applications that facilitate enhancing accessibility and quality of life. Yet, the lack of extensive datasets from disabled individuals hinders the development of tailored TMD systems. Our study assesses two data collection methods in TMD for disability research: using non-wheelchair users to simulate wheelchair activities (Simulation Real IMU) and generating synthetic sensor data from videos (Virtual IMU). Results show that, when using a larger dataset and multiple sensor modalities, models trained on Simulation Real IMU perform better. However, models trained on both Simulation Real IMU and Virtual IMU exhibited similar performances when sensors were restricted to accelerometer and gyroscope only. This finding guides future researchers toward the use of Simulation Real IMU for comprehensive, multimodal sensor studies, provided they have sufficient budget and time. However, the more cost and time-efficient Virtual IMU can be a viable alternative in scenarios using basic sensors.

Leng & Jung et al., Ubicomp 2024 (HASCA)

  • Title: Emotion Recognition on the Go: Utilizing Wearable IMUs for Personalized Emotion Recognition
  • Authors: Zikang Leng (Co-first Author), Myeongul Jung (Co-first Author), Sungjin Hwang, Seungwoo Oh, Lizhe Zhang, Thomas Plötz, Kwanguk Kim
  • Type of paper: HASCA Workshop Paper
  • Summary: In the field of emotion recognition, traditional methods often rely on motion capture technologies to recognize human emotions by analyzing body motion. However, these methods are privacy-intrusive and impractical for everyday use. To address the requirements of privacy and practicality, this paper develops a novel personalized Automatic Emotion Recognition (AER) system utilizing inertial measurement units (IMUs) embedded in common wearable devices. Our approach emphasizes personalization to adapt to cultural and individual variations in emotional expression. To reduce the amount of data that needs to be collected from users, we employ cross-modality transfer approaches. These allow us to generate virtual IMU data from established human motion datasets, such as Motion-X and Mocap, thus enriching our training set without extensive real-world data collection. By integrating this virtual IMU data with real IMU data collected from participants, we have developed a personalized wearable-based AER system that is both less intrusive and more practical for real-world applications.

Hiremath et al., Ubicomp 2024 (HASCA)

  • Paper Title:Game of LLMs: Discovering Structural Constructs in Activities using Large Language Models.
  • Authors: Shruthi Hiremath, Thomas Plötz
  • Type of paper: Hasca Workshop Paper
  • Summary: Human Activity Recognition is a time-series analysis problem. A popular analysis procedure used by the community assumes an optimal window length to design recognition pipelines. However, in the scenario of smart homes, where activities are of varying duration and frequency, the assumption of a constant sized window does not hold. Additionally, previous works have shown these activities to be made up of building blocks. We focus on identifying these underlying building blocks–structural constructs, with the use of large language models. Identifying these constructs can be beneficial especially in recognizing short-duration and infrequent activities. We also propose the development of an activity recognition procedure that uses these building blocks to model activities, thus helping the downstream task of activity monitoring in smart homes.

Haresamudram et al., Ubicomp 2024

  • Tutorial Title: Solving the Sensor-Based Activity Recognition Problem (SOAR): Self-Supervised, Multi-Modal Recognition of Activities from Wearable Sensors
  • Authors: Harish Haresamudram, Chi Ian Tang, Sungho Suh, Paul Lukowicz, Thomas Plötz
  • Type of paper: Tutorial
  • Summary: Feature extraction remains the core challenge in Human Activity Recognition (HAR) - the automated inference of activities being performed from sensor data. Over the past few years, the community has witnessed a shift from manual feature engineering using statistical metrics and distribution-based representations, to feature learning via neural networks. Particularly, self-supervised learning methods that leverage large-scale unlabeled data to train powerful feature extractors have gained significant traction, and various works have demonstrated its ability to train powerful feature extractors from large-scale unlabeled data. Recently, the advent of Large Language Models (LLMs) and multi-modal foundation models has unveiled a promising direction by leveraging well-understood data modalities. This tutorial focuses on existing representation learning works, from single-sensor approaches to cross-device and cross-modality pipelines. Furthermore, we will provide an overview of recent developments in multi-modal foundation models, which originated from language and vision learning, but have recently started incorporating inertial measurement units (IMU) and time-series data. This tutorial will offer an important forum for researchers in the mobile sensing community to discuss future research directions in representation learning for HAR, and in particular, to identify potential avenues to incorporate the latest advancements in multi-modal foundation models, aiming to finally solve the long-standing activity recognition problem.

Keynote

Thomas Ploetz will give a keynote at the WellComp2024 workshop