Towards Predicting Fragility in End-to-End Web Tests

Abstract

Automated end-to-end web tests are typically implemented as scripts that leverage dedicated libraries to simulate user interactions with web pages in a remotely controlled web browser. These tests are crucial for ensuring the functionality and reliability of web applications, as well as confirming non-regression on new releases. Still, as web applications evolve, maintaining the end-to-end test code is one of the main challenges faced by practitioners. Indeed, even minor alterations in web pages can easily break existing test code, rendering it unable to correctly locate and interact with web page elements. This issue is commonly known as web test fragility.

Many works in the literature have investigated the root causes of web test fragility and proposed approaches to repair these tests automatically or to produce more robust tests. In this short paper, we tackle the fragility problem from a different perspective. Rather than trying to repair broken tests, the proposal we are currently working on aims at estimating the extent to which a web test is prone to breakage, by considering both the source code of the web test and the web pages of the application under test. We envision that our proposal could offer substantial benefits to web testing processes, allowing testers to improve their test suites before breakages occur, saving time and money in the process. Preliminary experiments we conducted on an open-source web application showed that the computed fragility scores are highly correlated with test breakages in subsequent web application versions.

Data and other materials

A replication package to support the findings presented in this paper is available at