PerFail 2025

Fourth International Workshop on Negative Results in Pervasive Computing

March, 2025

Co-located with IEEE PerCom 2025 in Washington DC, USA

“Learn from the mistakes of others. You can’t live long enough to make them all yourself.” - Eleanor Roosevelt

ABOUT

Not all research leads to fruitful results, trying new ways or methods may surpass the state of the art, but sometimes the hypothesis is not proven or the improvement is insignificant. But failure to succeed is not failure to progress and this workshop aims to create a platform for sharing insights, experiences, and lessons learned when conducting research in the area of pervasive computing.

While the direct outcome of negative results might not contribute much to the field, the wisdom of hindsight could be a contribution itself, such that other researchers could avoid falling into similar pitfalls. We consider negative results to be studies that are run correctly (in the light of the current state of the art) and in good practice, but fail in terms of proving of the hypothesis or come up with no significance. The “badness” of the work can also come out as a properly but unfittingly designed data collection, or (non-trivial) lapses of hindsight especially in measurement studies.

We took the insights and discussion from last year and wrote a paper about the collected information. You can read the published manuscript in IEEE Pervasive Computitng here.

PerFail also has been featured in the Nature feature article "Illuminating 'the ugly side of science': fresh incentives for reporting negative results". You can read the article here.

CALL FOR PAPERS

The papers of this workshop should highlight lessons learned from the negative results. The main outcome of the workshop is to share experiences so that others avoid the pitfalls that the community generally overlooks in the final accepted publications. All areas of pervasive computing, networking and systems research are considered. While we take a very broad view of “negative results”, submissions based on opinions and non-fundamental circumstances (e.g. coding errors and “bugs”) are not in scope of the workshop as they do not indicate if the approach (or hypothesis) was bad.

The main topics of interests include (but are not limited to):

Studies with unconvincing results which could not be verified (e.g. due to lack of datasets)
Underperforming experiments due to oversights in system design, inadequate/misconfigured infrastructure, etc.
Research studies with setbacks resulting in lessons learnt and acquired hindsights (e.g. hypothesis with too limiting or too broad assumptions)
Unconventional, abnormal, or controversial results that contradict expectations of the community
Unexpected problems affecting publications, e.g. ethical concerns, institutional policy breaches, etc.
“Non-publishable” or “hard-to-publish” side-outcomes of the study, e.g . mis-trials of experiment methodology/design, preparations for proof-of-correctness of results, etc.

We also welcome submissions from experienced researchers that recounts post-mortem of experiments or research directions they have failed in the past (e.g. in a story-based format). With this workshop, our aim is to normalize the negative outcomes and inherent failures while conducting research in pervasive computing, systems and networking, and provide a complementary view to all the success stories in these fields.

Important Dates*

Paper Submission: ~~November 17~~ December 1, 2024 (extended)
Author Notification: ~~January 8~~ January 10, 2025 (extended)
Camera-ready Due: February 2, 2025
Workshop Date: March, 2025

* All dates are AoE (check it here).

SUBMISSION GUIDELINES

Regular papers should present novel perspectives within the scope of the workshop: negative results, lessons learned, and other fruitful “failure” stories. Papers must be in PDF format and contain 6 pages maximum (including references). Papers should contain names and affiliations of the authors (not blinded). All papers must be typeset in double-column IEEE format using 10pt fonts on US letter paper, with all fonts embedded. Submissions must be made via EasyChair. The IEEE LaTeX and Microsoft Word templates, as well as related information, can be found at the IEEE Computer Society website.

PerFail will be held in conjunction with IEEE Percom 2025. All accepted papers will be included in the Percom workshops proceedings and included and indexed in the IEEE digital library Xplore. At least one author will be required to have a full registration in the Percom 2025 conference and present the paper during the workshop in person. There will be no workshop-only registration.

Submission link: here

REGISTRATION

Each accepted workshop paper requires a full PerCom registration (no registration is available for workshops only). Otherwise, the paper will be withdrawn from publication. The authors of all accepted papers must guarantee that their paper will be presented at the workshop. Papers not presented at the workshop will be considered as a "no-show" and it will not be included in the proceedings.

Registration link: here

TECHNICAL PROGRAM

08:40 - 08:45

Opening Remarks

08:45 - 09:35

Keynote: Failing Without Being a Failure

Prof. Dr. Gregor Schiele (University of Duisburg-Essen, Germany)

Dr. Gregor Schiele is professor for embedded systems and leads since november 2014 the Intelligent Embedded Systems group at the University Duisburg-Essen at the campus Duisburg. Before that he was working from 2012 to 2014 at the Insight Centre for Data Analytics and the Digital Enterprise Research Institute (DERI) as well as at the National University of Ireland, Galway. From 2006 to 2012 he was working at the department of Prof. Dr. Christian Becker at the University Mannheim. He wrote his doctorate 2007 at the University Stuttgart at the department of Prof. Dr. Kurt Rothermel.

09:35 - 10:00

Paper Presentations

How to Minimize the Impact of Sociopolitical Factors on the Implementation of Pervasive Computing Projects

Authors: Mirosław Hajder, Mateusz Liput, Piotr Hajder, Lucyna Hajder, Mateusz Mojżeszko, Robert Rogólski and Łukasz Kiszkowiak

Read Abstract

The article discusses the influence of socio-political factors on the implementation of research projects in the area of pervasive computing, with particular emphasis on the instability of the client's behavior. The authors, based on their own experience, present the negative effects of changes in the client's expectations during the implementation of the project, emphasizing their impact on scientific research. They point to lobbying activities and financing problems as the main causes of instability, which leads to difficulties in achieving research goals and implementing innovative solutions. The authors analyze an example of a project to build a regional computer network in which a change in the client's expectations led to a modification of the design assumptions and the way the network is used. They indicate that the original concept of the network, based on the equal treatment of all entities, was changed to favor the telecom operators. The article draws attention to the need for stability and predictability in the implementation of research projects to avoid negative consequences for the development of science and technology.

10:00 - 10:30

Coffee Break

10:30 - 12:10

Paper Presentations

Motivation Matters: Challenges and Pit-Falls of Crowdsourced Online Studies

Authors: Eileen Becks, Viktor Matkovic and Torben Weis

Read Abstract

In the field of pervasive computing, conducting studies online via crowdsourcing platforms has become a widely adopted approach due to its advantages in efficiently accessing diverse, and scalable participant samples. However, not all study designs are equally suited for online implementation without compromising data quality. This issue became evident during our study on human interaction with AI assistance. By leveraging Prolific, a crowdsourcing website enabling online studies, we investigated how humans interact with AI-generated hints while solving a pipe maze game. The study employed ”Poor Man’s Eye Tracking,” combining mouse tracking with obscured vision fields, to monitor participant behaviour. As our hypothesis, rising diligence in checking AI hints, after the incorrectness of the AI is pointed out, was not confirmed, analysing our data suggests a lack of intrinsic motivation and attention among participants, as reflected in the low effort to verify AI hints. This falsified the reliability and validity of the collected data. Our study showed that even a careful pre-selection of participants through a crowdsourced website cannot prevent issues of low motivation and attention. Moreover, identifying missing motivation or inattentiveness in non-questionnaire components, such as game scenarios, requires additional data collection and analysis, such as mouse tracking, to retrospectively filter invalid datasets. This paper discusses potential causes and methods to mitigate these issues and proposes strategies to identify and prevent data distortion caused by missing motivation and inattentiveness of subjects.

Assessing the Impact of Sampling Irregularity in Time Series Data: Human Activity Recognition As A Case Study
Authors: Mengxi Liu, Daniel Geißler, Sizhen Bian, Bo Zhou and Paul Lukowicz

Read Abstract

Human activity recognition (HAR) ideally relies on data from wearable or environment-instrumented sensors sampled at regular intervals, enabling standard neural network models optimized for consistent time-series data as input. How- ever, real-world sensor data often exhibits irregular sampling due to, for example, hardware constraints, power-saving measures, or communication delays, posing challenges for deployed static HAR models. This study assesses the impact of sampling irregularities on HAR by simulating irregular data through two methods: introducing slight inconsistencies in sampling intervals (times- tamp variations) to mimic sensor jitter, and randomly removing data points (random dropout) to simulate missing values due to packet loss or sensor failure. We evaluate both discrete-time neural networks and continuous-time neural networks, which are designed to handle continuous-time data, on three public datasets. We demonstrate that timestamp variations do not significantly affect the performance of discrete-time neural networks, and the continuous-time neural network is also ineffective in addressing the challenges posed by irregular sampling, possibly due to limitations in modeling complex temporal patterns with missing data. Our findings underscore the necessity for new models or approaches that can robustly handle sampling irregularity in time-series data, like the reading in human activity recognition, paving the way for future research in this domain.

Collaborative Human Activity Recognition with Passive Inter-Body Electrostatic Field
Authors: Sizhen Bian, Vitor Fortes Rey, Siyu Yuan and Paul Lukowicz

Read Abstract

The passive body-area electrostatic field has recently been aspiringly explored for wearable motion sensing, harnessing its two thrilling characteristics: full-body motion sensitivity and environmental sensitivity, which potentially empowers human activity recognition both independently and jointly from a single sensing front-end and theoretically brings significant competition against traditional inertial sensor that is incapable in environmental variations sensing. While most works focus on exploring the electrostatic field of a single body as the target, this work, for the first time, quantitatively evaluates the mutual effect of inter-body electrostatic fields and its contribution to collaborative activity recognition. A wearable electrostatic field sensing front-end and wrist-worn prototypes are built, and a sixteen-hour, manually annotated dataset is collected, involving an experiment of manipulating objects both independently and collaboratively. A regression model is finally used to recognize the collaborative activities among users. Despite the theoretical advantages of the body electrostatic field, the recognition of both single and collaborative activities shows unanticipated less-competitive recognition performance compared with the accelerometer. However, it is worth mentioning that this novel sensing modality improves the recognition F-score of user collaboration by 16% in the fusion result of the two wearable motion sensing modalities, demonstrating the potential of bringing body electrostatic field as a complementary power-efficient signal for collaborative activity tracking using wearables.

Synthetic Sensory Data Generators: How Much Progress Are We Making?
Authors: Renuka Sharma, Abdelwahed Kamis, Sara Khalifa, Dan Bretherton and Brano Kusy

Read Abstract

Synthetic Sensory Data Generators (SSDGs) promise to advance the state of intelligent sensing by providing labeled training data at almost no cost. Such data can be used to train real-world sensory classification models without manual data collection and annotation. In this work, we dissect a promis- ing paradigm of SSDGs (based on human motion generation) and reveal a culprit that could hinder future progress. SSDGs are postfixed with a simple ”calibration” component; to bridge the distributional gap between real and synthetic data. In this study, we conduct a critical review of this component and analyses its contribution to the data synthesis pipeline. Our finding reveals that, without a proper understanding of the calibration process, the performance of SSDGs is often overestimated. We make a number of observations demonstrating that the performance of current SSDGs heavily depends on the calibration process. First, generating synthetic data without calibration leads to poorly performing down stream classifiers (when trained on synthetic data). Second, while calibration can be unsupervised, only supervised implementation is usable. This raises the question of whether SSDGs are better than the relatable few-shot learners that doesn’t require data synthesis effort. We advocate for fully unsupervised SSDGs. Third, in some cases, the calibration value outweighs that of the actual data generation process. Specifically, our experiments demonstrate that a classifier trained on random data is equally good to that trained on synthetic data when both are calibrated! Thus, downstream classification performance isn’t necessarily a good metric of the generated data quality. Our findings call for rethinking the current evaluation protocols of SSDGs.

12:10 - 13:30

Lunch

13:30 - 14:20

Keynote: Title will be added shortly

Prof. Kimberly Cornell (University of Albany, New York)

Kimberly Cornell joined the College of Emergency Preparedness, Homeland Security and Cybersecurity (CEHC) at the University at Albany in August 2022. Prior to joining CEHC, Dr. Cornell was a professor at the College of St. Rose in Albany, New York. As an associate professor in Computer Science, she has instructed at both the undergraduate and graduate levels in the fields of Computer Science, Information Technology, Computer Information Systems and Cybersecurity. Her research focuses on cryptographic protocol analysis and related areas, including indistinguishability. She will support the burgeoning research ecosystem at CEHC by hosting a cryptographic protocol analysis laboratory.

14:20 - 15:10

Paper Presentations

Human Factor as a Security Issue in ISP Networks
Authors: Marek Michalski

Read Abstract

This paper presents a real case in which a network user noticed a serious security issue and reported it to the ISP. Despite providing detailed information, the problem was not resolved as quickly as technically possible due to human factors. Fortunately, the issue was eventually recognized and resolved. It could have caused significant damage, but it is unknown whether it did. The paper highlights weaknesses in certain procedures that should be improved.

Unforeseen SILLY Errors in Network Simulations and Visualizations
Authors: Koojana Kuladinithi, Yevhenii Shudrenko, Aliyu Makama, Leonard Fisser and Andreas Timm-Giel

Read Abstract

Factors like incorrect parameterization, unsuitable KPIs, or overlooking key aspects in design and modeling may result in misleading conclusions, time- and resource-waste. In this paper, we share our genuine experiences from working with bachelor's, master's and PhD students, with the focus on three scenarios where distinct simulation errors commonly occur. The first two scenarios highlight cases of unjustifiable results, leading to investigations that uncovered "silly errors" in randomization and visualization. The third scenario, though properly modeled, failed to capture the overall picture needed to reflect real-world outcomes. By sharing these lessons learned across experience levels, we aim to help students and researchers achieve more credible results while saving valuable time.

15:10 - 15:35

Coffee Break

15:35 - 16:00

Paper Presentations

When Flow Balance Backfires: The Traffic Surge from Reflexive Forwarding in ICNs
Authors: Asanga Udugama, Sneha Kulkarni, Thenuka Karunathilake and Anna Förster

Read Abstract

Information-Centric Networking (ICN) is a promising networking architecture for the Internet of Things (IoT). Architectural elements such as in-network caching and content-based security can help communicate the vast amounts of content the IoT is expected to generate. Since the IoT exhibits several communication patterns distinct from the request-response interactions of ICNs, a draft standard called Reflexive Forwarding was discussed at the IETF. Reflexive Forwarding (RF) was proposed to address communications in the IoT, among other use cases. Our investigations, although initially showing promising results, revealed that RF, when used in large-scale IoT deployments, was associated with exponential growth in packet traffic. This work highlights this problem by examining various aspects of ICN operation when employing RF for communications in the IoT.

16:00 - 16:50

Keynote: Title will be added shortly

Prof. Sara Khalifa (Queensland University of Technology, Australia)

Sara Khalifa is an associate professor at Queensland University of Technology. Her research revolves around the broad aspects of ubiquitous sensing and edge computing for Internet of Things (IoT) applications. Specifically, she focuses on enhancing the energy efficiency of mobile sensing systems and developing lightweight machine learning techniques for resource-constrained sensing devices. From 2016-2023, she was with CSIRO’s Data61 establishing the foundational research area of “Energy Harvesting Sensing (EHS)” as a core research focus developing a new paradigm for energy-efficient sensing and context recognition, opening up a multitude of new applications, generating IP, and attracting significant funding and commercial interest.

16:50 - 17:00

Best Paper & Closing Words