Hello everyone. I'm Luigi
Libero Lucio Starace and in
this short video I'll present
ReCover: a curated dataset for
Regression Testing Research
Starting with a little bit of
context, regression testing is
the set of practices aimed
at gaining confidence that,
within software evolution, maintenance
changes to the code base
do not introduce faults in
previously-validated
functionalities. This
straightforward approach to
regression testing consists in
re-executing the entire test
suite on each new release.
This re-test all approach,
however, comes often short
when applied in modern
iterative development practices
due to time and resource
constraints. Hence more
refined and progression testing
techniques are required, and
designing such techniques is a
hot and longstanding topic in
the software engineering
community. To assess the
effectiveness of this
regression test optimization
techniques, proper empirical
evaluations need to be put in
place. In this context, data
sets can be seen as collections
of version pairs in which the
older version works fine. So
we have a green bar and all
test pass, while one or more
faults have been introduced in
the more recent version.
A number of less recent works
used datasets with artificially
injected faults. More recently,
Thanks also to the efforts of
the Mining Software
Repositories Community,
datasets with real regression
faults (and RTPTorrent is
a good example) have been
collected. So why are we
proposing yet another dataset?
Well, in our experience in
regression testing research, we
found that existing datasets
are affected by some
limitations. Firstly,
most datasets, for the sake of
compactness, do not include
coverage reports nor the
original source code, which are
both crucial to a number of
regression test optimization
techniques. So researchers
need to retrieve the source
code and to compute code
coverage themselves.
Retrieving the source code. However,
is not always easy. We found
that existing datasets often
include outdated references to
GitHub repositories that have
been moved or to build logs in
continuous integration
platforms such as TravisCI that
have expired and are simply no
longer available. Lastly,
even when it is possible to
retrieve the source code,
building the software projects
is not always straightforward.
Projects require different
versions of Java, different
libraries, different
configurations. So a lot of
manual work is still required
to build the projects and to
compute the coverage reports. To
address these limitations, We
propose to ReCover. ReCover is
a dataset designed for
regression testing research and
features 114 version pairs from
28 heterogeneous Java Maven
projects. Each version pair
includes at least one real
fault in the more recent version,
which is to say at least one
test fails. Complete, buildable
source code of both versions.
Fine-grained test execution and
test coverage reports, computed
with the state-of-the-art tool Open Clover
And it all comes
within a Docker container so
that everyone and everywhere
can effortlessly replicate the
build process without the
hassle of dealing with
missing dependencies or misconfigurations
We built
ReCover by automatically
mining TravisCI and GitHub,
with a tool we designed. Our
tool, which is available upon
request and can be easily
extended to support additional
data sources, consists in three
key components. The data
provider is responsible for
selecting which software
projects and versions should be
considered for mining. Then
the job processor component is
the one that actually does the
heavy lifting. It downloads
the sources of the software
projects by interacting with
TravisCI and GitHub and then
automatically builds the
software and instruments it to
compute test coverage. And
lastly, the reporter component
produces summaries of the mining
activities in SQLite and XML
format. So
if you are working with
regression testing
optimization techniques and
need a dataset which provides,
out of the box, full source code
including real-world faults,
fine-grained test coverage
reports at statement-level,
and the Docker environment for
hassle-free replication of
the build process, check out
ReCover. And if you have
any questions, drop us an
email. Bye!