PerFail 2024

Third International Workshop on Negative Results in Pervasive Computing

15 March, 2024

Co-located with IEEE PerCom 2024, Biarritz, France

“Learn from the mistakes of others. You can’t live long enough to make them all yourself.” - Eleanor Roosevelt


Not all research leads to fruitful results, trying new ways or methods may surpass the state of the art, but sometimes the hypothesis is not proven or the improvement is insignificant. But failure to succeed is not failure to progress and this workshop aims to create a platform for sharing insights, experiences, and lessons learned when conducting research in the area of pervasive computing.

While the direct outcome of negative results might not contribute much to the field, the wisdom of hindsight could be a contribution itself, such that other researchers could avoid falling into similar pitfalls. We consider negative results to be studies that are run correctly (in the light of the current state of the art) and in good practice, but fail in terms of proving of the hypothesis or come up with no significance. The “badness” of the work can also come out as a properly but unfittingly designed data collection, or (non-trivial) lapses of hindsight especially in measurement studies.

We took the insights and discussion from last year and wrote a paper about the collected information. You can read the published manuscript in IEEE Pervasive Computitng here.


The papers of this workshop should highlight lessons learned from the negative results. The main outcome of the workshop is to share experiences so that others avoid the pitfalls that the community generally overlooks in the final accepted publications. All areas of pervasive computing, networking and systems research are considered. While we take a very broad view of “negative results”, submissions based on opinions and non-fundamental circumstances (e.g. coding errors and “bugs”) are not in scope of the workshop as they do not indicate if the approach (or hypothesis) was bad.

The main topics of interests include (but are not limited to):

  1. Studies with unconvincing results which could not be verified (e.g. due to lack of datasets)
  2. Underperforming experiments due to oversights in system design, inadequate/misconfigured infrastructure, etc.
  3. Research studies with setbacks resulting in lessons learnt and acquired hindsights (e.g. hypothesis with too limiting or too broad assumptions)
  4. Unconventional, abnormal, or controversial results that contradict expectations of the community
  5. Unexpected problems affecting publications, e.g. ethical concerns, institutional policy breaches, etc.
  6. “Non-publishable” or “hard-to-publish” side-outcomes of the study, e.g . mis-trials of experiment methodology/design, preparations for proof-of-correctness of results, etc.

We also welcome submissions from experienced researchers that recounts post-mortem of experiments or research directions they have failed in the past (e.g. in a story-based format). With this workshop, our aim is to normalize the negative outcomes and inherent failures while conducting research in pervasive computing, systems and networking, and provide a complementary view to all the success stories in these fields.

Important Dates

Paper submission: November 17th, 2023 (extended to December 1st, 2023)
Author notification: January 8th, 2024
Camera-ready due: February 2nd, 2024
Workshop date: March 15th, 2024


Regular papers should present novel perspectives within the scope of the workshop: negative results, lessons learned, and other fruitful “failure” stories. Papers must be in PDF format and contain 6 pages maximum (including references). Papers should contain names and affiliations of the authors (not blinded). All papers must be typeset in double-column IEEE format using 10pt fonts on US letter paper, with all fonts embedded. Submissions must be made via EDAS. The IEEE LaTeX and Microsoft Word templates, as well as related information, can be found at the IEEE Computer Society website.

PerFail will be held in conjunction with IEEE Percom 2024. All accepted papers will be included in the Percom workshops proceedings and included and indexed in the IEEE digital library Xplore. At least one author will be required to have a full registration in the Percom 2024 conference and present the paper during the workshop (either remotely or in location). There will be no workshop-only registration.

Submission link: HERE


Each accepted workshop paper requires a full PerCom registration (no registration is available for workshops only). Otherwise, the paper will be withdrawn from publication. The authors of all accepted papers must guarantee that their paper will be presented at the workshop. Papers not presented at the workshop will be considered as a "no-show" and it will not be included in the proceedings.

Registration link: here


Welcome and Opening Remarks: 08:30 - 08:35

Peter Zdankin (University of Duisburg-Essen)

Technical Session 1: 08:35 - 09:50

Session Chair: Peter Zdankin (University of Duisburg-Essen)

Context Matters: Lessons Learned from Emulated and Simulated TSN Environments
Authors: Filip Rezabek, Marcin Bosk, Leander Seidlitz, Jörg Ott, Georg Carle
Clock synchronization precision and security are imperative in modern Time Sensitive Networking (TSN) systems and applications building on top of them. Their deployment validation is challenging due to many, sometimes incompatible or inaccessible, implementations of TSN and related standards. Such aspects make ensuring their proper functionality and performance crucial and require thorough evaluation and validation to ensure system's robustness. For that reason, we consider various platforms using hardware or simulations as an evaluation tool. This work shows the lessons learned during real-world deployments of the Precision Time Protocol (PTP) and MACsec. As a part of that, we identify challenges of combining various sensors and how to possibly overcome them using open-source solutions. To protect sensitive traffic of PTP and other TSN traffic, we evaluate synergies between MACsec and its performance implications. The findings point to requirements for future deployments of TSN systems. Last, we discuss simulation as a supporting tool for TSN experimentation and its limitations. Nevertheless, we highlight how different evaluation approaches can provide a complete view of the system under test.
Designing and Executing a Large-scale Real-life Affective Study
Authors: Joanna Komoszyńska, Dominika Kunc, Bartosz Perz, Adam Hebko, Przemysław Kazienko and Stanislaw Saganowski
Despite advancements in emotion recognition, the suitability of existing AI models for daily use remains uncertain. Creating more accurate solutions requires amounts of data uncollectible in a laboratory setting, thus forcing researchers to focus on an information-rich environment: everyday life. However, the constraints of in-the-field studies are different than those of in-the-laboratory ones, which are familiar to most researchers. In this work, we delve into the challenges in designing and conducting a large-scale real-life affective study, employing wearable devices and smartphones for data collection. We highlight encountered issues, made decisions, and mistakes, making our experiences valuable for researchers aiming to conduct studies in a real-life setting.
Too Good To Be True: accuracy overestimation in (re)current practices for Human Activity Recognition
Authors: Andrés Tello, Victoria Degeler, Alexander Lazovik
Today, there are standard and well established procedures within the Human Activity Recognition (HAR) pipeline. However, some of these conventional approaches lead to accuracy overestimation. In particular, sliding windows for data segmentation followed by standard random k-fold cross validation, produce biased results. An analysis of previous literature and present-day studies, surprisingly, shows that these are common approaches in state-of-the-art studies on HAR. It is important to raise awareness in the scientific community about this problem, whose negative effects are being overlooked. Otherwise, publications of biased results lead to papers that report lower accuracies, with correct unbiased methods, harder to publish. Several experiments with different types of datasets and different types of classification models allow us to exhibit the problem and show it persists independently of the method or dataset.


09:50 - 10:30

Technical Session 2: 10:30 - 11:45

Session Chair: Malte Josten (University of Duisburg-Essen)

Can we generate real faces from rPPG signals? Probably not
Authors: Li Honghan, Nhi Nguyen, Constantino Alvarez Casado, Xiaoting Wu, Miguel Bordallo Lopez
The potential of generating authentic human facial images from remote photoplethysmography (rPPG) signals is a compelling idea, with significant implications for biometric authentication and human-computer interaction. This study explores it by using a large-scale dataset to train a diffusion-based generative model, leveraging rPPG signals extracted from facial videos. The initial training phase yields promising results, with the model demonstrating a capacity to synthesize facial likenesses that closely match the corresponding subjects in the training dataset. However, the performance notably falters during validation with an independent dataset, where a marked divergence between generated and actual faces becomes apparent. A subsequent human perception study corroborates this discrepancy. These observations suggest that rPPG signals alone may not be reliable for accurately generating realistic facial imagery.
Do Weak Brain Signals Get Amplified When Strong Brain Signals are Evoked?
Authors: Ekansh Gupta, Cheng-Yeh Chen and Raghupathy Sivakumar
Brain-computer interfaces (BCIs) facilitate an unprecedented fusion between the human mind and pervasive computing systems, enabling users to engage with connected devices in their environment through neural signaling. Despite their potential, BCIs face certain challenges that hinder their widespread proliferation, such as low SNR and high noise levels in brain signals recorded via non-invasive techniques like EEG, high variability in signals among users that hinders generalization, usability challenges, etc. While brain signals like the error potential (ErrP) showcase low SNR and have lower detection accuracy, there are other kinds of signals that showcase high detection rates and resilience to noise. Motivated by this disparity, we ask ourselves if the abstract cognitive states involved in the evocation of such resilient signals be leveraged to amplify or augment the weaker signals, and thus provide them a performance boost. We investigate this hypothesis by designing an experiment to interface these two kinds of signals and collect EEG data in our lab through human trials. We evaluate our hypothesis and contrast our results with other datasets of isolated signals using spatial filtering and deep learning models. We obtain negative results and reflect on the insights and the lessons learned based on them and talk about plausible explanations and future work while also reassessing our initial hypothesis.
Echocardiographic Epicardial Adipose Tissue Quantification: Challenges and Insights
Authors: Payel Patra, Andrea Bianchi, Daniele Di Pompeo, Antinisca Di Marco
e-Health applications, as a cornerstone of modern distributed systems, must synergize with advanced analysis methodologies, incorporating image processing, statistical, and predictive techniques to expedite diagnosis and optimize therapeutic strategies. Cardiovascular disease (CVD) presents a formidable health challenge, claiming 18 million lives annually, with projections set to worsen due to population aging, the rise of metabolic diseases, and gaps in effective prevention and precise risk stratification. A pivotal indicator of cardiovascular health, the epicardial adipose tissue (EAT) thickness, is traditionally estimated by medical professionals without a standardized and precise procedure. This paper chronicles our endeavor to automate the delineation of EAT from echocardiogram videos, a fundamental precursor to its thickness quantification. We confronted the intricate task of interpreting echocardiographic data and trialed a variety of image processing methods aimed at clarifying the EAT's representation amidst the heart's dynamic activity and inherent imaging noise. Our study's narrative contributes to the pervasive computing domain, envisaging the deployment of such medical applications as on-demand cloud services for medical experts and institutions, thus fostering collaborative, efficient, and accurate cardiovascular health real-time assessment. Unfortunately, our study failed and in this paper we analyse the reasons and we report the lesson learned.

Keynote: 11:45 - 12:30

Title: Why Tasklets have not taken over the world (yet)?

Session Chair: Ella Peltonen (University of Oulu)



Prof. Dr. Janick Edinger
University of Hamburg, Germany

(A brief description will follow shortly)

Best Paper Award and Final Words: 12:30 - 12:35

Session Chair: TBD


Organizing Committee


Nitinder Mohan Technical University of Munich


Ella Peltonen University of Oulu


Peter Zdankin University of Duisburg-Essen


Tanya Shreedhar University of Edinburgh


Malte Josten University of Duisburg-Essen

Technical Program Committee


Aaron Ding TU Delft


Daniela Nicklas University of Bamberg


Eirini Eleni Tsiropoulou University of New Mexico


Gregor Schiele University of Duisburg-Essen


Gürkan Solmaz NEC Labs Europe


Javier Berrocal Universidad de Extremadura


Jörg Ott Technical University of Munich


Jon Crowcroft University of Cambridge


Jussi Kangasharju University of Helsinki


Oliver Gasser MPI-Informatics


Ralph Holz University of Münster


Roman Kolcun University of Cambridge


Sandip Chakraborty Indian Institute of Technology


Simone Ferlin Red Hat


Suzan Bayhan University of Twente


Stephan Sigg Aalto University


Torben Weis University of Duisburg-Essen

Previous Editions