PerFail 2024

Third International Workshop on Negative Results in Pervasive Computing

15 March, 2024

Co-located with IEEE PerCom 2024, Biarritz, France

“Learn from the mistakes of others. You can’t live long enough to make them all yourself.” - Eleanor Roosevelt

ABOUT

Not all research leads to fruitful results, trying new ways or methods may surpass the state of the art, but sometimes the hypothesis is not proven or the improvement is insignificant. But failure to succeed is not failure to progress and this workshop aims to create a platform for sharing insights, experiences, and lessons learned when conducting research in the area of pervasive computing.

While the direct outcome of negative results might not contribute much to the field, the wisdom of hindsight could be a contribution itself, such that other researchers could avoid falling into similar pitfalls. We consider negative results to be studies that are run correctly (in the light of the current state of the art) and in good practice, but fail in terms of proving of the hypothesis or come up with no significance. The “badness” of the work can also come out as a properly but unfittingly designed data collection, or (non-trivial) lapses of hindsight especially in measurement studies.

We took the insights and discussion from last year and wrote a paper about the collected information. You can read the published manuscript in IEEE Pervasive Computitng here.

CALL FOR PAPERS

The papers of this workshop should highlight lessons learned from the negative results. The main outcome of the workshop is to share experiences so that others avoid the pitfalls that the community generally overlooks in the final accepted publications. All areas of pervasive computing, networking and systems research are considered. While we take a very broad view of “negative results”, submissions based on opinions and non-fundamental circumstances (e.g. coding errors and “bugs”) are not in scope of the workshop as they do not indicate if the approach (or hypothesis) was bad.

The main topics of interests include (but are not limited to):

  1. Studies with unconvincing results which could not be verified (e.g. due to lack of datasets)
  2. Underperforming experiments due to oversights in system design, inadequate/misconfigured infrastructure, etc.
  3. Research studies with setbacks resulting in lessons learnt and acquired hindsights (e.g. hypothesis with too limiting or too broad assumptions)
  4. Unconventional, abnormal, or controversial results that contradict expectations of the community
  5. Unexpected problems affecting publications, e.g. ethical concerns, institutional policy breaches, etc.
  6. “Non-publishable” or “hard-to-publish” side-outcomes of the study, e.g . mis-trials of experiment methodology/design, preparations for proof-of-correctness of results, etc.

We also welcome submissions from experienced researchers that recounts post-mortem of experiments or research directions they have failed in the past (e.g. in a story-based format). With this workshop, our aim is to normalize the negative outcomes and inherent failures while conducting research in pervasive computing, systems and networking, and provide a complementary view to all the success stories in these fields.

Important Dates

Paper submission: November 17th, 2023 (extended to December 1st, 2023)
Author notification: January 8th, 2024
Camera-ready due: February 2nd, 2024
Workshop date: March 15th, 2024

SUBMISSION GUIDELINES

Regular papers should present novel perspectives within the scope of the workshop: negative results, lessons learned, and other fruitful “failure” stories. Papers must be in PDF format and contain 6 pages maximum (including references). Papers should contain names and affiliations of the authors (not blinded). All papers must be typeset in double-column IEEE format using 10pt fonts on US letter paper, with all fonts embedded. Submissions must be made via EDAS. The IEEE LaTeX and Microsoft Word templates, as well as related information, can be found at the IEEE Computer Society website.

PerFail will be held in conjunction with IEEE Percom 2024. All accepted papers will be included in the Percom workshops proceedings and included and indexed in the IEEE digital library Xplore. At least one author will be required to have a full registration in the Percom 2024 conference and present the paper during the workshop (either remotely or in location). There will be no workshop-only registration.

Submission link: HERE

REGISTRATION

Each accepted workshop paper requires a full PerCom registration (no registration is available for workshops only). Otherwise, the paper will be withdrawn from publication. The authors of all accepted papers must guarantee that their paper will be presented at the workshop. Papers not presented at the workshop will be considered as a "no-show" and it will not be included in the proceedings.

Registration link: here

TECHNICAL PROGRAM

Welcome and Opening Remarks: 08:30 - 08:35

Peter Zdankin (University of Duisburg-Essen)


Technical Session 1: 08:35 - 09:50

Session Chair: Peter Zdankin (University of Duisburg-Essen)

Context Matters: Lessons Learned from Emulated and Simulated TSN Environments
Authors: Filip Rezabek, Marcin Bosk, Leander Seidlitz, Jörg Ott, Georg Carle
Clock synchronization precision and security are imperative in modern Time Sensitive Networking (TSN) systems and applications building on top of them. Their deployment validation is challenging due to many, sometimes incompatible or inaccessible, implementations of TSN and related standards. Such aspects make ensuring their proper functionality and performance crucial and require thorough evaluation and validation to ensure system's robustness. For that reason, we consider various platforms using hardware or simulations as an evaluation tool. This work shows the lessons learned during real-world deployments of the Precision Time Protocol (PTP) and MACsec. As a part of that, we identify challenges of combining various sensors and how to possibly overcome them using open-source solutions. To protect sensitive traffic of PTP and other TSN traffic, we evaluate synergies between MACsec and its performance implications. The findings point to requirements for future deployments of TSN systems. Last, we discuss simulation as a supporting tool for TSN experimentation and its limitations. Nevertheless, we highlight how different evaluation approaches can provide a complete view of the system under test.
Designing and Executing a Large-scale Real-life Affective Study
Authors: Joanna Komoszyńska, Dominika Kunc, Bartosz Perz, Adam Hebko, Przemysław Kazienko and Stanislaw Saganowski
Despite advancements in emotion recognition, the suitability of existing AI models for daily use remains uncertain. Creating more accurate solutions requires amounts of data uncollectible in a laboratory setting, thus forcing researchers to focus on an information-rich environment: everyday life. However, the constraints of in-the-field studies are different than those of in-the-laboratory ones, which are familiar to most researchers. In this work, we delve into the challenges in designing and conducting a large-scale real-life affective study, employing wearable devices and smartphones for data collection. We highlight encountered issues, made decisions, and mistakes, making our experiences valuable for researchers aiming to conduct studies in a real-life setting.
Too Good To Be True: accuracy overestimation in (re)current practices for Human Activity Recognition
Authors: Andrés Tello, Victoria Degeler, Alexander Lazovik
Today, there are standard and well established procedures within the Human Activity Recognition (HAR) pipeline. However, some of these conventional approaches lead to accuracy overestimation. In particular, sliding windows for data segmentation followed by standard random k-fold cross validation, produce biased results. An analysis of previous literature and present-day studies, surprisingly, shows that these are common approaches in state-of-the-art studies on HAR. It is important to raise awareness in the scientific community about this problem, whose negative effects are being overlooked. Otherwise, publications of biased results lead to papers that report lower accuracies, with correct unbiased methods, harder to publish. Several experiments with different types of datasets and different types of classification models allow us to exhibit the problem and show it persists independently of the method or dataset.

Break

09:50 - 10:30


Technical Session 2: 10:30 - 11:45

Session Chair: Malte Josten (University of Duisburg-Essen)

Can we generate real faces from rPPG signals? Probably not
Authors: Li Honghan, Nhi Nguyen, Constantino Alvarez Casado, Xiaoting Wu, Miguel Bordallo Lopez
The potential of generating authentic human facial images from remote photoplethysmography (rPPG) signals is a compelling idea, with significant implications for biometric authentication and human-computer interaction. This study explores it by using a large-scale dataset to train a diffusion-based generative model, leveraging rPPG signals extracted from facial videos. The initial training phase yields promising results, with the model demonstrating a capacity to synthesize facial likenesses that closely match the corresponding subjects in the training dataset. However, the performance notably falters during validation with an independent dataset, where a marked divergence between generated and actual faces becomes apparent. A subsequent human perception study corroborates this discrepancy. These observations suggest that rPPG signals alone may not be reliable for accurately generating realistic facial imagery.
Do Weak Brain Signals Get Amplified When Strong Brain Signals are Evoked?
Authors: Ekansh Gupta, Cheng-Yeh Chen and Raghupathy Sivakumar
Brain-computer interfaces (BCIs) facilitate an unprecedented fusion between the human mind and pervasive computing systems, enabling users to engage with connected devices in their environment through neural signaling. Despite their potential, BCIs face certain challenges that hinder their widespread proliferation, such as low SNR and high noise levels in brain signals recorded via non-invasive techniques like EEG, high variability in signals among users that hinders generalization, usability challenges, etc. While brain signals like the error potential (ErrP) showcase low SNR and have lower detection accuracy, there are other kinds of signals that showcase high detection rates and resilience to noise. Motivated by this disparity, we ask ourselves if the abstract cognitive states involved in the evocation of such resilient signals be leveraged to amplify or augment the weaker signals, and thus provide them a performance boost. We investigate this hypothesis by designing an experiment to interface these two kinds of signals and collect EEG data in our lab through human trials. We evaluate our hypothesis and contrast our results with other datasets of isolated signals using spatial filtering and deep learning models. We obtain negative results and reflect on the insights and the lessons learned based on them and talk about plausible explanations and future work while also reassessing our initial hypothesis.
Echocardiographic Epicardial Adipose Tissue Quantification: Challenges and Insights
Authors: Payel Patra, Andrea Bianchi, Daniele Di Pompeo, Antinisca Di Marco
e-Health applications, as a cornerstone of modern distributed systems, must synergize with advanced analysis methodologies, incorporating image processing, statistical, and predictive techniques to expedite diagnosis and optimize therapeutic strategies. Cardiovascular disease (CVD) presents a formidable health challenge, claiming 18 million lives annually, with projections set to worsen due to population aging, the rise of metabolic diseases, and gaps in effective prevention and precise risk stratification. A pivotal indicator of cardiovascular health, the epicardial adipose tissue (EAT) thickness, is traditionally estimated by medical professionals without a standardized and precise procedure. This paper chronicles our endeavor to automate the delineation of EAT from echocardiogram videos, a fundamental precursor to its thickness quantification. We confronted the intricate task of interpreting echocardiographic data and trialed a variety of image processing methods aimed at clarifying the EAT's representation amidst the heart's dynamic activity and inherent imaging noise. Our study's narrative contributes to the pervasive computing domain, envisaging the deployment of such medical applications as on-demand cloud services for medical experts and institutions, thus fostering collaborative, efficient, and accurate cardiovascular health real-time assessment. Unfortunately, our study failed and in this paper we analyse the reasons and we report the lesson learned.

Keynote: 11:45 - 12:30

Title: Why Tasklets have not taken over the world (yet)?

Session Chair: Ella Peltonen (University of Oulu)

Speaker

Image

Prof. Dr. Janick Edinger
University of Hamburg, Germany

(A brief description will follow shortly)


Best Paper Award and Final Words: 12:30 - 12:35

Session Chair: TBD

COMMITTEE

Organizing Committee

Image

Nitinder Mohan Technical University of Munich

Image

Ella Peltonen University of Oulu

Image

Peter Zdankin University of Duisburg-Essen

Image

Tanya Shreedhar University of Edinburgh

Image

Malte Josten University of Duisburg-Essen

Technical Program Committee

Image

Aaron Ding TU Delft

Image

Daniela Nicklas University of Bamberg

Image

Eirini Eleni Tsiropoulou University of New Mexico

Image

Gregor Schiele University of Duisburg-Essen

Image

Gürkan Solmaz NEC Labs Europe

Image

Javier Berrocal Universidad de Extremadura

Image

Jörg Ott Technical University of Munich

Image

Jon Crowcroft University of Cambridge

Image

Jussi Kangasharju University of Helsinki

Image

Oliver Gasser MPI-Informatics

Image

Ralph Holz University of Münster

Image

Roman Kolcun University of Cambridge

Image

Sandip Chakraborty Indian Institute of Technology

Image

Simone Ferlin Red Hat

Image

Suzan Bayhan University of Twente

Image

Stephan Sigg Aalto University

Image

Torben Weis University of Duisburg-Essen

Previous Editions